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I know of no other movement in American higher educa- 
tion in the past decade that has spurred more conversation, more 
consternation, and more concern than that of assessment. Many 
of us who began our careers prior to the assessment movement 
in the 1990s have been compelled to learn new skills, engage in 
assessment activities, appreciate the benefits of quality assess- 
ment, and embrace assessment as a central element of our work. 
Our younger colleagues who entered the higher education arena 
more recently have also quickly come to know that an under- 
standing of assessment is a fundamental key to effectiveness, 
institutionalization, and longevity on a program level, as well 
as a career-building asset on a personal level. As director of 
the National Resource Center for The First- Year Experience and 
Students in Transition, it is my pleasure to introduce this timely 
monograph. 

The very existence of the National Resource Center is due 
in large part to the foresight of the central administration at the 
University of South Carolina more than a quarter-century ago, 
who insisted that an experimental course undergo rigorous 
assessment. The result of these assessment efforts was the in- 
stitutionalization, growth, and development of the university's 
first-year seminar. University 101. John N. Gardner, the course 
director, and his staff founded the National Resource Center in 
1986 as an outgrowth of this academic department. 

Collaboration with the Policy Center on the First Year of 
College has made this monograph possible, as their work focuses 
directly on the improvement of the first college year and the de- 
velopment and dissemination of a range of first-year assessment 
procedures and tools that can be used to strengthen or confirm 
practices in the curriculum, co-curriculum, and institutional 
policy. This monograph is but one of many collaborative projects 
by our two centers, whose missions both have the improvement 
of undergraduate education at their cores. 

The editor, Randy Swing, opens the monograph with a co- 
gent overview of first-year assessment. In this primer, he makes 
a strong case for the importance of assessment by outlining the 
purposes of assessment and describing the qualities of effective 
assessment in a clear and compelling manner. Practical tips on 
first steps for getting started in assessment along with a discus- 
sion of pitfalls to avoid, help to demystify assessment for even 
the most apprehensive practitioner. His introduction continues 
with a discussion of four assessment structures, the two forms 
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of assessment data, and the myriad sources of assessment data available from which those 
engaging in assessment can choose. He closes the introductory chapter with a discussion 
of strategies for organizing for assessment that will be useful to all, from those engaged in 
assessment for the first time to seasoned institutional researchers. 

The balance of this valuable monograph builds on these assessment basics and includes 
essays and case studies on a variety of noteworthy assessment approaches and topics writ- 
ten by researchers, scholars, and practitioners representing a wide range of institutional 
settings, allied organizations, and commercial survey instrument developers. The essays are 
creative and thought provoking and provide a basis for fertile discussions on the countless 
possibilities for campus-based assessment approaches. A typology of assessment instru- 
ments concludes the monograph and enables readers to make an uncomplicated comparison 
of many of the commercially available assessment surveys. 

I am convinced that this monograph will provide assessment novices and seasoned 
assessment directors alike a new resource that gathers together in a single volume a wealth 
of ideas on approaches to assessment and detailed information on available assessment 
tools. All of us at the National Resource Center on The First- Year Experience and Students 
in Transition at the University of South Carolina and the Policy Center on the First Year of 
College involved with this project sincerely hope this is so. Assessment of our efforts on 
behalf of undergraduate students is incredibly important. It is not easy, but it is worth- 
while. Not only is assessment critical to the future of our programs and demanded by our 
institutions and external accrediting bodies, but it can also both help us confirm that our 
efforts are worthwhile and assist us in continuing to improve what we do. 

Best wishes in your endeavors. 

Mary Stuart Hunter, Director 

National Resource Center for The First- Year Experience and Students in Transition 
University of South Carolina 
February 2004 




Increasing enrollments, reductions in funding for higher 
education, and greater diversity of new students continue to 
shape the work of instructors and administrators of first-year 
students at American colleges and universities. Taking pride in 
"flunking out" large numbers of students is a distant memory 
on most campuses. Instead, institutions of higher education 
have increasingly embraced their obligation for assisting stu- 
dents with the transition to the college learning environment. 
Over the past two decades nearly every campus in America has 
launched initiatives specifically designed to improve the suc- 
cess of new students. It might be said that the challenge of the 
last two decades of the 20th century was to disseminate good 
practices in first-year education and to encourage institutions to 
embrace their responsibilities for the success of their students. 
That challenge has, in large measure, been accomplished. The 
challenges facing educators in the first decades of the new cen- 
tury are to perfect those good practices launched over the last 
decades, adjust practices to accommodate a more diverse student 
population, and increase the cost efficiencies of services to new 
students. These new challenges call for greater knowledge about 
assessment practices. It is simply not enough to collect anecdotal 
stories of student success — the refinement of good practices re- 
quires systematic and ongoing evaluation of outcomes. 

The First- Year Assessment Listserv (FYA-L) is one way 
that educators are developing their knowledge of assessment 
tools and techniques. The idea of a listserv focused on first- 
year assessment was conceived by John N. Gardner and Betsy 
O. Barefoot as part of a grant application to The Pew Charitable 
Trusts. That proposal outlined a new organization intended to 
extend the work of the National Resource Center on The First- 
Year Experience and Students in Transition with a new focus on 
assessment as a means to improving the first college year. 

The task of establishing the proposed listserv fell to me 
early in 2000. I accepted the challenge with mixed feelings as it 
was clear that educators were hungry for help with assessment 
but, it was also clear that many listservs fail to do more than fill 
subscribers' inboxes with disorganized and unhelpful chatter. I 
decided that an effective listserv needed to deliver meaningful 
content to busy educators as well as provide opportunities for 
peer-to-peer sharing of ideas and concerns. 

FYA-L was launched in early 2000 with a promise that 
subscribers would receive, at least twice a month, an invited. 
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edited essay from an assessment professional or practitioner on a topic relevant to assess- 
ing the first college year. Undergirding the concept for these invited essays was a simple 
philosophy — busy educators could benefit from a brief essay, about 1,000 words, focused 
on a single "good" assessment idea that could be read and understood in five minutes or 
less. The essays would be delivered via e-mail and also stored on a web site to enhance 
the ability of subscribers to retrieve ideas later or to share them easily with others. 

This new concept of a listserv as a kind of online magazine caught on. More than 
300 subscribers signed on during the first three days of the listserv. The number rose to 
more than 600 subscribers by the end of the first month and topped 1,200 subscribers within 
a few months. FYA-L grew to have the largest number of subscribers of any listserv on 
the topic of higher education assessment — even though the list continued to specialize in 
assessment topics focused on the first year of college. Those initial essays were collected in 
a monograph. Proving and Improving: Strategies for Assessing the First College Year, published 
in 2001 by the National Resource Center at the University of South Carolina. 

While that first monograph was moving to press, FYA-L continued to deliver invited 
essays focused on assessment tools and techniques. With the assistance of my colleagues 
at the Policy Center — Betsy O. Barefoot, Marc Cutright, John N. Gardner, Samantha Land- 
grover, Michael J. Siegel, and Angie Whiteside — additional invited essays were collected, 
edited, and shared with subscribers. FYA-L continues to operate as a joint venture of the 
Policy Center and the National Resource Center under the editorial leadership of Michael 
J. Siegel and Steven W. Schwartz. 

Essays from FYA-L along with a number of new contributions comprise this sec- 
ond monograph. Proving and Improving Volume II: Tools and Techniques for Assessing the First 
College Year. 1 Written for educators with little or moderate experience assessing first-year 
students and initiatives the monograph provides an overview of assessment and specific 
ideas for a variety of assessment activities. The monograph also includes a detailed typology 
of first-year assessment instruments which will be helpful to even experienced assessment 
practitioners. The monograph includes the following sections. 

• "An Introduction to First- Year Assessment" provides a primer on assessment 
that identifies good practices, steps, and structures for starting an assessment 
initiative and tips for organizing assessment efforts. 

• Part I is built on the premise of using existing data before collecting new data 
and focuses on how institutional records can be used in first-year assess- 
ment. 

• Part II provides advice about methods for collecting, analyzing, and using 
the "student voice" — captured through student interviews or journals — in 
first-year assessment. 

• Part III focuses on methods to evaluate a first-year initiative using focus groups 
or classroom environment inventories. 

• Part IV contains essays about a variety of specific assessment survey instru- 
ments. 

• Part V contains a discussion of instruments and structures designed to examine 
cognitive development in students. 

• Part VI offers essays on instruments designed to examine specific student 
traits. 

• Appendix A provides a typology of instruments commonly used in first-year 
assessment efforts. The typology includes contact information and is cross- 
referenced to essays in this monograph. 
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Whether read from cover to cover or used as a reference guide to assessment instru- 
ments, Volume II provides a wide array of ideas and resources for gaining new perspectives 
on the first college year. These essays share a common element — they are presented not as 
a call to conduct assessment for assessment's sake but rather for assessment to be a means 
to the greater end of improving the first year of college for all new students. 

Randy L. Swing, Co-Director and Senior Scholar 
Policy Center on the First Year of College 
Brevard College 
February 2004 



Notes 

1 Special thanks are extended to Tracy L. Skipper, Editorial Projects Coordinator at the Na- 
tional Resource Center for The First- Year Experience and Students in Transition. Tracy's 
editorial skills greatly improved every essay in this monograph. She created the organiza- 
tional structure for the monograph and oversaw the full production process. Simply stated, 
this monograph was guided and moved forward at every step of the process by Tracy. 
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The "accountability movement" in American higher educa- 
tion, and around the world, has challenged educators to "prove" 



that our efforts matter — that our educational goals are advanced 
by the work that we do. Many educators now find their job de- 
scription includes "assessing learning outcomes" or "measuring 
institutional effectiveness." Rather than being an unwelcome in- 
trusion, most educators are embracing assessment efforts because 
such efforts dovetail nicely with professional curiosity about effec- 
tive educational practices and commitment to first-year students. 
Still, few educators have received adequate preparation for roles 
in assessment through traditional disciplinary or student affairs 
graduate programs. 

This chapter, and indeed this entire monograph, is intended to 
provide assistance to educators charged with assessment respon- 
sibilities related to the first year of college. Moreover, this chapter 
serves as a primer — an overview — of assessment structures and 
decisions. In essence, this opening section answers the question, 
"How would you know high-quality assessment of the first year 
if you saw it?" 

In a nutshell, high-quality assessment always leads to one of 
two outcomes: It either creates improvement or confirms existing 
practice. Notice what was not mentioned. Production of reams 
of "interesting data" or the identification of problems outside of 
institutional control (e.g., poverty, changes in demographics) are 
not hallmarks of effective assessment. Stated in other words, as- 
sessment is not an end in itself but always a means to one of two 
desirable ends: (a) improvement of individual or program per- 
formance or (b) confirmation of existing practice. The measure of 
high-quality assessment, then, is best established by how, or if, the 
assessment results are used. It is not enough simply to fill book 
shelves with unread reports and undigested data even if these were 
produced by methodologically perfect assessment practices. 

Proving the efficacy of existing structures and increasing de- 
sired individual or program-level outcomes through improvement 
of existing practices or development of new practices are equally 
valued outcomes of effective assessment. Whether the intention is 
to "prove or improve," assessment does not start with the assump- 
tion that something is wrong or that the purpose of assessment 
is to find problem areas. Rather, the opening assumption holds 
that it is as important to know what is working well, so those 
components can be protected and supported, as it is to find areas 
where improvement is needed. 
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Qualities of Effective Assessment 

The good news is that much has been learned over the past two decades about how 
to structure assessment activities that are likely to result in useful outcomes. High-quality 
assessment: 

• Focuses on what matters most. There are always numerous "interesting questions," 
but assessment is not simply an intellectual exercise. High-quality assessment 
springs from an organization's well-defined goals, objectives, and mission. 
Clarity about desired outcomes is the first step in successful assessment. 

• Focuses on elements that the organization can change. Little can be gained by 
studying fixed characteristics or conditions that are exceedingly difficult to 
change. 

• Is built on the goodwill of participants and stakeholders. Assessment efforts that 
are properly timed, orchestrated, and explained to participants are more likely 
to produce trustworthy data and outcomes. 

• Is multidimensional. Assessment seldom meets the rigorous standards of con- 
trolled experimentation, but it can produce credible results when information 
is established through corroborating studies using multiple measures and 
methods. 

• Includes input from all stakeholders. Externally mandated and contrived assess- 
ment is less likely to produce meaningful change than assessment created by 
and for those directly involved in the assessed activity. 

• Places findings in an appropriate context. Comparative benchmarks, longitudinal 
data, and/ or professional judgments are needed to provide context for assess- 
ment findings. 

• Produces comprehensible results. Assessment reports are written in accessible 
language for each target audience. A general rule of thumb is that aggregated 
data may be widely disseminated, but data that potentially could identify a 
specific individual should be treated with the highest level of confidentiality 
and disseminated with great care. Reports should be written at the appropriate 
level of specificity for each target audience (e.g., executive summary, concise 
edition, full report). 

• Is disseminated and used. Results must be shared with the individuals who shape 
the desired outcome, and decision makers should cite the use of assessment 
data that inform campus decisions. 

Getting Started 

There are plenty of potential potholes in the assessment planning phase but most 
will create no more than a minor bump in the road to a successfully completed assess- 
ment. One notable exception is the very serious calamity known as "assessment paraly- 
sis," the decision that nothing can be done because the perfect assessment plan cannot 
be established. Every assessment plan has limitations and any perfectly controlled 
assessment, if such even exists, is unlikely to be "doable" in a real world environment. 
The key to success is to acknowledge the limitations of assessment methods and either 
accept them or design multiple assessment activities that have different limitations. It 
is quite common for assessment plans to evolve over time rather than be fully formed 
when initially launched, so even a partial outline may be sufficient to start. 
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This advice is not intended to suggest that sloppy assessment structures are to be 
tolerated; rather, the recommendation is to start simple and grow the complexity of the 
assessment efforts over time. When starting a new assessment program, a number of very 
basic questions simply will not have been studied. Targeting those most doable issues first 
allows the complexity of assessment efforts to evolve as the institution's assessment skills 
develop. 

A second potential pitfall is failing to establish whether the purpose of the assessment 
is to evaluate individuals or programs. Begin with the notion that every assessment effort 
is intended to evaluate either individual-level or program-level outcomes. In both cases, 
data will be collected from students or about students, but in the former, each student's 
data stands alone, while in the latter, student data are aggregated to establish the "average" 
experience of students with a program, course, event, or structure. The difference is huge. 
When assessing individual students, the results must be very accurate — especially if the 
results have consequences for the student such as placement in or out of special courses or 
admission to a program/ major. When measuring attributes of individuals, it is imperative 
that the assessment tools be proven to be reliable (i.e., having little arbitrary variance so 
that about the same outcome would be expected if an individual were tested twice). Reli- 
able assessment instruments are time consuming to construct and should be undertaken 
only with adequate lead time and significant resources. Development of individual-level 
assessment tools typically should be left to measurement and testing professionals. Given 
the cost of time and resources, institutions should look for existing commercially available 
instruments before undertaking development of new instruments. 

Ideally, program-level assessments would also be built from highly reliable instru- 
ments, but when data are aggregated a degree of unexplainable variance will have little 
effect on the assessment result. For every individual where the score is a little high, there 
will be another individual where the score is a little low so the results balance out in the 
aggregate. In other words, program-level assessment can be reasonably achieved with 
instruments that are not as "perfect" as those required when performing individual-level 
assessment. 

In reality, individual-level assessments can be effectively aggregated to form program- 
level assessment. For example, a highly reliable exit assessment that determines whether 
individuals "pass" or "fail" could also be used to evaluate the overall effectiveness of a 
program for all participants by aggregating the individual results. The problem occurs 
when assessment moves in the opposite direction — from program-level to individual-level. 
Analysis of program-level assessment, for example, may reveal which students scored lower 
than others on some outcome measure such as writing, computing, or public speaking. The 
temptation is to "remediate" those lower scoring students. Great caution is needed when 
switching from evaluating programs to evaluating individuals unless the reliability of the 
instrument used is well-documented and the potential use of the result was explained to 
participants prior to the measurement. If students will be individually evaluated, they must 
be told about the purposes and consequences of the assessment before data are collected. 

Assessment Structures 

Four forms of assessment (i.e., criterion referenced, value added, benchmarking, and 
prediction) undergird the majority of all higher education assessment activities. While a full 
review of assessment structures is beyond the scope of this chapter, a basic understanding 
of these four assessment structures will guide many of the decisions needed in developing 
an assessment plan. 
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Criterion Referenced 

Criterion-referenced assessment structures evaluate individuals or units against an 
established criterion, often called a "cut score" or "minimum score." A criterion-referenced 
assessment commonly used in the first year of college is the writing placement test that has 
one or more pre-established "passing" scores. The score on the test might, for example, 
determine the examinee's placement in a developmental, first-level, or advanced English 
course. The purpose of criterion-referenced assessment is to determine if individuals or 
programs are above or below an established level set for passing. 

Criterion-referenced assessments provide valuable information about the level of 
outcome achieved. However, they are an incomplete assessment model because they pro- 
vide very limited information, if any at all, to support change or confirm which practices 
contributed to the outcomes. Generally, such assessments are better suited to point out 
shortcomings of existing programs or individual achievement than to provide methods 
for improvement when outcomes fall short of expectations. 

Value-Added Assessment 

The structure which has become the "gold standard" in assessment is the value-added 
model created by Astin (1991). The concept appears quite simple at first, but the complexity 
becomes apparent as the model is used to assess first-year students and programs. 

Value-added assessment depends on three pieces of data: input, environment, and 
outcomes. Input variables represent the baseline or conditions before "treatment." In 
first-year assessment that means knowing about the skills, knowledge, attitudes, or char- 
acteristics that students possess as they enter college. Input measures may reflect admis- 
sions decisions, but generally do not reflect impact of the college on students because they 
are taken before the student has experienced significant interaction with the institution. In 
other words, an institution that recruits a "better" set of students cannot be said to have 
created those "better" students simply based on higher levels of desirable measures at the 
point of entry into college. 

Environment variables record what students do while enrolled in the first year of col- 
lege. Some environmental measures are simply dichotomies such as whether a student did 
or did not enroll in a particular course or attend some student support initiative. Other 
measures are continuous variables, meaning they are measures of degree or frequency, 
such as how many hours per week a student works for pay or studies. Measurements of 
environment variables are necessary because all students, even if they attend the same 
institution, have varying patterns of engagement with the institution. Even when student 
participation is required, the level of engagement is likely to vary greatly so it is imperative 
to establish how much students participated in the intended educational experience as part 
of evaluating the experience. For example, as any instructor of a first-year seminar can at- 
test, some students who enroll in a seminar actively engage in activities, while others only 
show up for class occasionally! Clearly it cannot be said that both kinds of students received 
the same educational experience though both are on the same class roster. Environment 
measures establish what students do, individually or collectively. These measures may 
focus on educational practices designed by institutions of higher education or on student 
choices for how they spend their time outside formal educational structures. 

Outcome measures are collected at the end of educational experiences (i.e., summative 
measures) or at any point after an educational experience begins (i.e., formative measures). 
Outcome measures should closely align with the goals and objectives of the educational 
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intervention to be evaluated. While unintended outcomes are likely and the discovery of 
those may have significance, the purpose of first-year assessment is normally focused on 
determining the level at which specific and intentional outcomes occurred. In general, it 
is not efficient to toss a broad net to "discover" what outcomes developed. Focusing on 
a limited set of desired outcomes is more likely to produce information that can guide 
program improvement or confirm existing practices. 

These three data points, I, E, and O, become an effective assessment model when 
inputs and environments can explain differences in net outcomes (i.e., observed outcomes 
less the base-line measurement of that outcome). Perhaps the best way to explain the I- 
E-O assessment model is to explore how incomplete models fail to be valuable. Below are 
examples of four incomplete I-E-O models (or how NOT to do assessment!) 

1 . An outcomes-only model, as the name indicates, is limited to measures of out- 
comes only. The model does not contain information about entry-level char- 
acteristics or what environments students participate in during the first year. 

Only the level of achievement for a specific outcome is known. Such models 
lack the evidence needed to link outcomes to specific college experiences or to 
show that outcomes are greater than they were at the point of entry. As such, 
the model cannot confirm current practice or suggest how current practice 
could be improved. 

2. An environment-only model can produce interesting findings that describe stu- 
dent experiences. It has been popular, for example, to count the percentage 
of students who consume alcohol in the first year, attend tutoring, or enroll in 
a first-year seminar. These data are simply counts of actions. In the absence 
of input or outcome data they are not very helpful in understanding how 
experiences impact students. When the environment-only data are collected 
as continuous variables they might take on the appearance of greater useful- 
ness. For example, an assessment of the study patterns of students might ask 
students to report how many hours per week they study. The results can be 
reported as an array (e.g., 5% studied one hour per week, 7% studied two hours 
per week, 1% studied 20 hours per week, and so on). The assumption that 
more is better or that each level marks a significant difference from the level 
immediately above or below makes an environment-only continuous variable 
model appear to be effective, but in the absence of outcome data such conclu- 
sions must be made with great caution as research does not always support 
intuitively "obvious" conclusions. 

3. An input-outcome model can be very appealing because it does produce some 
valuable information and answers the question, "Did students change dur- 
ing the first year?" In essence, this model provides a pre-test and post-test of 
variables of interest so it can be determined if the selected variable increased, 
decreased, or remained stable across the first year. The model is incomplete, 
however, because it lacks information about what occurred between the two 
measurement points that could have had an impact on the desired outcome. 

This model may be useful in pointing out that growth did or did not happen, 
but it would not make clear which existing practices were effective / ineffective 
or which practices should be continued or improved. 

4. Environment-outcome models are frequently misused in higher education assess- 
ment. A classic example is comparing students who enroll in a learning com- 
munities program with those who do not enroll (environment) on measures 
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such as persistence, writing skill, or grade-point average (outcomes). It is 
exciting to discover that participants in an intervention have higher outcomes 
than non-participants, but in reality the model fails in the absence of input 
data that would eliminate the alternative explanations that (a) the group in the 
intervention entered with different potential for success from the start and (b) 
the outcome was already present at the start and no change actually occurred. 

The environment-outcome model cannot confirm existing practice nor guide 
improvement because the impact of the intervention cannot be isolated from 
potentially confounding input variables. 

Unlike the incomplete models noted above, when all three data points (i.e., input, 
environment, and outcome) are present, the I-E-O model presents the best opportunity for 
confirming existing practice or guiding change. The results of such assessment are easy to 
explain because they show how characteristics present at entry, combined with collegiate 
experiences, produce intended outcomes. The model also allows for very complex analyses 
of the interactions of multiple variables and outcomes. 

Benchmarking 

Benchmarking structures are used to contextualize findings so that they are inter- 
pretable. Once an outcome has been measured the next question is "Does the measure 
indicate more or less achievement than would have been commonly expected?" One 
way to answer that question is to compare local outcomes to those of other organizations 
using the same or similar measurements. For example, we know that first- to second- 
year persistence rates vary by level of institutional selectivity and are influenced by 
the presence of interventions such as first-year seminars and learning communities. A 
benchmarking assessment of student persistence can be used to compare the persistence 
data from several institutions that recruit similar types of first-year students and offer 
similar first-year structures. 

Benchmarking is a variation of the I-E-O model wherein only the outcomes are 
measured. Input and environment measurements are not used directly, but only in the 
process of selecting peer comparisons (i.e., other units within an institution or similar 
"other" institutions). The selection of appropriate peers should ensure that the input 
variables and the environment variables are likely to be very similar, or in the language 
of statisticians, the I and E are "controlled for" or "held constant" so that the focus is only 
on changes in the O. For example, an institution with an average SAT of 1100 would first 
develop a list of other institutions with similar average SATs. The list would be refined 
by subsequently selecting only institutions offering similar first-year initiatives. The list 
could be further narrowed by restricting selection to institutions in the same geographic 
region, Carnegie classification, or some other useful criterion. 

Benchmarking explains how outcomes can vary when similar services are provided 
to similar students by evaluating differences in the quality of the environments that con- 
tributed to the outcomes. For example, no differences in outcomes would be expected 
when comparing two similarly situated first-year seminars enrolling similarly situated 
students if the two courses were of similar quality. If differences are found, then the next 
step in assessment would be to investigate the quality of the seminar delivery. Were 
teachers at the higher performing institutions trained more effectively? Did they use 
better teaching techniques? Were they more committed to their students, or otherwise 
delivering a more powerful intervention? 
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An alternative to peer benchmarking is to compare outcomes with aspirant peers (i.e., 
places that the local institution/ unit would like to emulate). By benchmarking against aspirant 
peers an organization can estimate the gains in outcomes that might occur if they emulated 
what the comparison group is already doing. 

Prediction Assessment 

The last of the four basic assessment structures focuses on the prediction of future out- 
comes. These assessments are a variation on the I-E-O model in that measured input and 
environment variables pass through a statistical modeling procedure to predict the level of 
outcome that would be expected if no additional intervention occurred. This form of assess- 
ment is frequently used in the first year to identify "high-risk" students before some undesirable 
outcome occurs (e.g., low GPA, dropping out, judicial action). In the first stage of such assess- 
ment, historical data are used to establish a model based on known inputs, environments, and 
outcomes. Local prediction models can be developed (e.g., institutional research officers on 
many campuses are experienced in this kind of statistical modeling), but it is more common 
for these models to be developed through national, multi-institutional research efforts. 

Prediction models can produce high-quality assessment outcomes by confirming that 
existing practices are effective or by showing where improvements might be made. These 
forms of assessment pose an ethical dilemma between focus on the assessment as a research 
initiative or improvement activity. It is rare, and ethically challenging, for institutions to assess 
the likelihood of a negative outcome occurring and then do nothing to attempt to change the 
outcome. If interventions are attempted and the ultimate outcome is different than the pre- 
dicted outcome it becomes impossible to determine if the intervention caused the improved 
outcome to be different than the prediction or if the outcome was inaccurately predicted 
by the model. Acting on results created by prediction models compromises the ability to 
"prove" the model's accuracy but provides opportunity to change, and hopefully improve, 
the outcome for students immediately. It is not a difficult choice for most educators working 
with first-year students to care more about helping every student succeed than statistically 
proving a theoretical model. 



Forms of Assessment Data 

A common misconception is that the terms "qualitative" and "quantitative" refer to 
different assessment structures. These terms define types of data, not assessment structures. 
Quantitative data are usually number-based and can be analyzed with statistical techniques. 
Qualitative data are usually word-based, although art and other forms of expression can 
also be used. Qualitative data are analyzed with special computer software or data reduction 
techniques that do not initially rely on statistical comparisons. In general, quantitative data 
are used to find statistical relationships between variables, and qualitative data are used to 
find themes and non-statistical relationships among variables. 

One of the principles of high-quality assessment suggested earlier in this chapter was 
the use of multiple views and multiple data sources. Rarely will either qualitative or quanti- 
tative data be adequate if used alone. Traditional research has relied mainly on quantitative 
data, but advances in research methodology and computing technology have increased the 
acceptance of qualitative analyses in higher education assessment. Any of the four assess- 
ment structures presented in the prior section can use either quantitative or qualitative data, 
and all are strengthened when both quantitative and qualitative data are employed in the 
assessment design. 
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Sources of Assessment Data 

Perhaps the best advice anyone undertaking a new assessment activity can receive — 
and the advice that is so rarely taken — is to use existing data before deciding to collect 
new forms of assessment evidence. Because institutions of higher education are large and 
loosely connected structures, it is unlikely that anyone on campus, including the registrar, 
institutional research officer, or director of administrative computing, knows about all the 
data sources that exist on a campus. Time spent exploring data sources may uncover an 
array of data that could be used to investigate first-year initiatives or be useful in conducting 
assessment of first-year students. The National Center for Higher Education Management 
Systems (NCHEMS) and the Policy Center on the First Year of College have produced a 
guide for an extensive "audit" of first-year data sources that could be used as a first process 
in a systematic assessment of the first year. NCHEMS's Data Audit Toolkit, described in 
detail by Karen Paulson later in this monograph, helps expand the concept of data sources 
that could be useful in studying the first year of college. For example, a computerized list 
of students and parents who attended new student orientation could be viewed as useful 
only in printing name tags, without acknowledgement that the same list could be used as 
a data source to determine if attendance at an orientation session contributed to any other 
first-year outcome of interest. The Data Audit Toolkit procedure helps institutions find "hid- 
den" data sources and evaluate the trustworthiness of each data source. In the example 
above, it would be important to know when the orientation list was created and if it was 
updated to remove the names of people who registered but did not attend. There will likely 
be some limitations anytime data collected for one purpose are subsequently used for a 
different purpose. The savings in time and costs may offset these limitations. 

Many sources of data exist for use in first-year assessment activities. The following 
descriptions suggest some of these sources. 

Institutional Records 

Many institutional research offices have a plethora of data on first-year students and 
initiatives. If the campus receives federally funded student financial aid, it is mandated to 
prepare reports for the Integrated Postsecondary Education Data System (IPEDS), which 
contains data about student enrollments, faculty characteristics, institutional resources, and 
more. IPEDS records are public data sources so there should be little difficulty in accessing 
aggregated data from these reports. The IR office typically is the source of a number of 
other reports to college guidebooks, news magazines, and national research efforts. The 
first stop in a new assessment effort should be the campus's institutional research office to 
determine what data are already available. 

An equally important source of data is the registrar/ student records office. Student 
grades, courses taken, enrollment terms, and more are standard records in the registrar's 
office. Because many of the data sources in the registrar's office are protected by the Fam- 
ily Educational Records and Privacy Act (FERPA), researchers will need to qualify to use 
these records. In general, institutions are free to use student records for purposes that 
improve educational practices. To ensure compliance with FERPA and other guidelines 
for the protection of human subjects, most institutions of higher education have created 
a formal structure, the Institutional Review Board (IRB), for certifying the use of campus 
data. It is common practice for the IRB to review an application for use of existing data or 
collection of new data to be used in campus assessment activities. Generally, use of data 
for assessment purposes are considered "exempt" from IRB oversight, but that status will 
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be determined by the IRB review process. Exempt simply means that use of the data 
does not constitute a significant threat to privacy or rights of human subjects and, as 
such, is exempt from oversight by the IRB. 

If use of the data is not determined to be "exempt," an IRB may still approve the 
study if certain conditions are met. For example, an IRB might stipulate rules for securely 
storing data, a schedule for destroying data upon the conclusion of their approved use, 
or that participation in a study be voluntary. Assessment practitioners must think ahead 
when applying for IRB approval as the conditions of the review might be different if 
the assessment results will be shared off campus, such as at a first-year conference or 
in an external publication. The IRB application should include notice of the possibility 
that the results will also be used as externally disseminated scholarship. 

Student Interviews 

It may be deceptively simple, but the best source of data is likely first-year stu- 
dents themselves. Student interviews are one way to capture this data. The downside 
of interviews is that data collection can be time consuming. Focus groups, a way to 
interview a number of students at one time, have the advantage of producing data from 
more students than individual interviews. The decision to conduct group or individual 
interviews should be guided by the type of information desired. Individual interviews 
are normally used to explore sensitive or private issues or when the evaluator is inter- 
ested in very detailed reports from a small number of individuals. More information 
about traditional focus groups is provided by Libby Morris' essay in this monograph. 

Student Writing 

Student journals, one-minute papers (see Angelo & Cross, 1993), and course embed- 
ded assignments are similar to student interviews but can be readily collected from a 
large sample of students. The essay by Elizabeth Hodges and Jean Yerian in this mono- 
graph about the Prompts Project at Virginia Commonwealth is an excellent example 
of using student writing for assessment purposes. The use of portfolios and electronic 
journals has made the collection and analyses of writing-based data more efficient. 

Surveys 

The advent of inexpensive photocopying, bubble-form publishing, and computer- 
ized data scanning equipment make paper and pencil surveys the major source of data 
in higher education assessment. In recent years, the paperless survey or web-based 
survey has reduced costs of data collection significantly and proven to be popular with 
students. There are numerous books available on designing and administering sur- 
veys, so this section will focus instead on the content areas of commercially available 
surveys that are commonly used to assess first-year initiatives and first-year students. 
These categories correspond with the typology of instruments located at the end of this 
monograph. 

Pre-enrollment/baseline data. These surveys are administered near the end of high 
school, during the admissions process, during new student orientation, or at any point 
prior to significant experience with an institution of higher education. Survey partici- 
pants report their expectations, impressions, goals, or hopes for the college experience, 
or they report their pre-enrollment behaviors and experiences. Data from these surveys 



do not reflect impact of the campus on students as they will have had only limited exposure 
to the campus prior to completing these surveys. Instead, these surveys can 

• Provide baseline data, telling us who our students are at the point of entry 

• Form gain scores when matched with post-tests 

• Provide co-variates and controls for advanced statistical evaluations 

• Assess the needs of new students 

• Monitor change in entering student characteristics if repeated over time 

Surveys of outcomes and experiences in the first college year. These surveys are designed 
for use near the end of the first college year and collect a multi-perspective report about col- 
legiate experiences. There are only three national instruments designed for use at the end 
of the first year, and each has been developed within the last five years as part of The Pew 
Charitable Trusts' accountability agenda for higher education. The intended timing of these 
surveys is their common link; otherwise, the three instruments are very different in that one 
is designed just for use in community colleges, one for use at four-year institutions only, and 
the third for use at both two- and four-year institutions. Two of the instruments (one for 
two-year institutions and one for four-year institutions) focus on student engagement with 
higher education in terms of how students spend their time. The other instrument collects 
a wide array of student opinions, beliefs, and behaviors. One instrument is designed to as- 
sess both first-year and senior students; another is designed as a post-test when combined 
with a matching survey at the college entry point, and the third survey is not specifically 
designed as a repeat measure. 

General surveys of student behavior, attitudes, study skills, satisfaction, and experiences. These 
surveys take a holistic approach by collecting information on a variety of college experiences 
and environments. While there is some overlap in content for these instruments, they each 
focus on a unique aspect of the first year and factor analyses can be used to group items into 
higher order constructs. Examples of topics included on these surveys are 

• "Average time" spent on academic and co-curricular tasks 

• Frequency of contact with peers, faculty, and staff 

• Self-reported gains in knowledge and self-confidence 

• Self-reported levels of knowledge and academic skills 

• Study skills such as time-management and note-taking 

• Satisfaction with college 

• Alcohol /drug use 

• Life management skills (e.g., relationship with roommate, parents, partners) 

Surveys of specific services/units/programs. These surveys deeply investigate a particular 
slice of the college experience with a series of narrowly drawn and specific questions about 
the full range of a given service, unit, or program. These instruments are built from items 
similar to those listed in the previous section. Instruments may include demographic and 
self-report questions so that results can be disaggregated by sub-populations. Areas with 
specialized survey instruments include: 

• Academic advising 

• Residence life 

• Campus student unions 

• First-year seminars 
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Surveys of specific populations. These surveys provide information to evaluate the 
experiences, satisfaction, or behaviors of a specific group of students. These instruments 
are not created exclusively for use in the first year but contain demographic variables 
so that first-year student responses can be selected from a population including upper- 
class students. Survey items are similar to those in the two preceding sections. It is the 
focus on a specific group, cohort, or sub-population which sets these instruments apart. 
Examples of sub-populations include: 

• Adult learners 

• Fraternity or sorority members 

• Non-returning students 

Placement and academic knowledge survey si tests. These instruments are designed to 
test academic knowledge and skills. Unlike opinion and satisfaction surveys, these in- 
struments usually have a right answer and the student is judged on his/her ability to 
select the best (right) answer. Some instruments contain a mix of discipline topics, but 
it is more common for tests to be designed to measure one specific knowledge domain. 
The use of these instruments may vary depending on the timing of the test. Measure- 
ments taken during new-student orientations are often designed to place students in 
the appropriate level of college courses based on knowledge at the point of admissions. 
Measurements taken at the end of the first year or in later years may be used to evaluate 
curricula or as individual assessments that serve as entrance requirements for a special 
major/ department. Examples of instrument focus include: 

• Writing 

• Critical thinking 

• Mathematics reasoning 

• General education humanities 

• Biological science 

End of program/ course evaluations. Several commercially available instruments have 
been designed for course evaluation. These instruments can be part of an assessment 
strategy, especially as a way to evaluate a program or campus-wide initiative. They are 
problematic for two reasons. First, the purpose of assessment is not to evaluate indi- 
vidual instructors, so any use of course evaluations that contain data about individual 
teachers raises the possibility that the data might be misused. Course evaluations used 
in the aggregate, however, can constitute effective evaluation of first-year initiatives. A 
second limitation in using course evaluations is that these instruments rarely contain 
demographic questions about respondents, so the lack of ability to merge these data with 
existing student records or to disaggregate these by sub-populations limits the usefulness 
of these data. Examples of items include: 

• Evaluation of course textbook(s) 

• Evaluation of teacher's classroom presence and effectiveness 

• Evaluation of teacher's out-of-class availability 

• Evaluation of course content 

• Evaluation of perceived fairness in grading 
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Organizing for Assessment 

A first-year assessment activity may occur in a single classroom and involve only a 
dozen students, in a large lecture hall with hundreds of students, or via web-based or e-mail 
survey with an infinite number of students. Assessment may be focused on any subgroup 
from a cohort residing in the same residence hall to large groups who share some other 
common characteristic. Many first-year assessment activities involve all or nearly all stu- 
dents who entered college in the same academic term and so may involve from a hundred 
to several thousand students. Whether a large-scale assessment that requires months of 
preparation or a small-scale assessment that can be arranged in just a few minutes, some 
basic organizing principles exist to ensure that the way the assessment is conducted does 
not unduly bias the results. 

1. Involve students. The good will of students undergirds successful assessment. 
Bombarding students with surveys and tests is likely to discourage their 
participation, reduce the level of effort given, and spoil reliability of the mea- 
sures. Assessment designs that judiciously use student time are more likely 
to produce appropriately high student effort and thus produce trustworthy 
data. Institutions are responsible for building and maintaining the good will 
of students toward assessment. Using focus groups, having students proctor 
surveys, asking students to help interpret results, letting students know how 
the institution uses assessment outcomes, and avoiding too many surveys are 
all ways to honor the involvement of students both as subjects and participants 
in assessment efforts. 

2. Use sampling. Often, little is gained by surveying every student. A carefully 
controlled sampling procedure will produce the same results as a population 
study. A limitation is that samples cannot always be disaggregated to study 
every sub-population (i.e., the numbers become too small when the sample 
is divided into many sub-groups). 

3. Divide and conquer. Multiple opportunities exist for data collection. For exam- 
ple, institutions might collect one survey during summer orientation, another 
during move-in day, and another at the first class. Ideally, assessment should 
be thought of as a long-term, on-going collection of events rather than as a 
one-time event. 

4. Use existing data whenever possible. A data audit often reveals that institutions 
hold great quantities of useable data that can be connected in a central dataset. 
Whenever possible, student identifiers should be used to link datasets — es- 
pecially to make use of demographic data already on file. Many assessment 
efforts start with time-consuming data collection and bog down during the 
data analysis phase. Existing data reduces collection time and increases time 
available to disaggregate, analyze, and report findings. 

5. Embed assessment in courses. Student time is used wisely and many motiva- 
tional issues are resolved when "regular" course exams and assignments are 
also used to create data for assessment purposes. Since course enrollments 
are not usually the result of random assignment, these data must be reviewed 
to determine the impact of self-selection on them. 

6. Use assessment days. Dedicating a day or half-day for assessment activities 
separate from class and orientation activities is another strategy for maximizing 
the students' and institution's time. The likelihood of student participation 
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can be increased by linking assessment activities to a reward, such as partici- 
pation in early registration for the next academic term. 

7. Sample classes - not students. An alternative to sampling the student popu- 
lation for a program-wide or campus-wide assessment is to sample the 
population of classes. A random or stratified sample of courses /class 
times will capture a diverse sample of students. A team of peer assistants, 
who visit the selected classes, can administer the surveys. Clearly, this 
strategy requires support from faculty and advance planning. 

8. Use cohort-based assessment cycles. Every year's entering class can be seen as 
a cohort to be surveyed on one topic at various points in their educational 
experience. For example, the assessment effort might focus on topic X 
for the entering class of 2005 and then on topic Y for the entering class of 
2006. Over time, each cohort adds to the institutional knowledge base. 
This concept reduces the data overload often produced by assessment 
efforts. It may, however, result in some limitations due to cross-sectional 
methodologies. 

9. Pay attention to response rates. Assessment built on a low-response rate 
is ripe for criticism and dismissal of the findings. Use a sample size 
and method that you can control, follow-up, and "work" to accomplish 
response rates above 50% of the sample. No matter the response rate, 
the researcher must examine the sample to determine if respondents are 
representative of the population of interest. A high response rate can still 
contain a systematic bias, while a low response rate could be "random" 
and highly representative. Researchers should look for evidence that the 
sample is representative on gender, race, and other characteristics. 

10. Signal the importance of assessment. Researchers should not assume that 
students or faculty know why the institution is conducting assessment. 
Using the appropriate mix of seriousness and fun, holding students ac- 
countable, and intentionally establishing a culture of assessment are 
strategies for underscoring the importance of assessment. 

Bringing it All Together 



The preceding "primer" highlights key decisions and opportunities in devel- 
oping assessment activities aimed at proving or improving the first year of college. 
Such efforts might produce greater resource allocations or increased respect for work 
with first-year students, but the main reason for undertaking assessment is to learn 
how best to achieve the goals and hopes for success for every first-year student. 
While there are many "right" ways to develop the information needed for long-term 
planning and implementation of effective first-year efforts, the most essential step in 
assessment is to begin! Our first-year students deserve no less than our professional 
curiosity and willingness to challenge ourselves to constantly improve the first year 
of college. 
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Introducing The Data Audit and Analysis Toolkit 



Karen Paulson The Data Audit and Analysis Project is a collaborative ef- 

fort between The Policy Center on the First Year of College and 
the National Center for Higher Education Management Systems 
(NCHEMS) designed to assist institutions in local data source 
compilation, data assembly, and data analysis procedures. The 
basic objective of a "data audit" as it relates to the first year of 
college is to identify and inventory data sources and needs within 
an institution so that a range of analyses about the first year can 
be conducted. In short, a data audit allows an institution to pe- 
riodically and systematically take stock of, and then mobilize, its 
data resources. The project resulted in The Data Audit and Analysis 
Toolkit, a comprehensive collection of materials, resources, and 
guidelines, and the focus of this essay. 

This essay builds on an earlier essay by my colleague, Peter 
Ewell (2001), entitled "Data Audit." He emphasized how the 
use of assessment in the first year of college could be a model 
for an institution creating a "culture of evidence." Assessment 
often begins with taking stock of data that already exists. His 
initial thinking provided the impetus for the Data Audit Toolkit, 
a project funded by The Pew Charitable Trusts and The Atlantic 
Philanthropies. 

In a pilot study, 10 institutions tested the functionality of 
data audit tools and materials and assessed the key factors dur- 
ing implementation of the data audit process. The pilot group 
included three technical or community colleges, two private 
institutions, and five public institutions of varying sizes and 
structures. The institutions were Augustana College (Illinois), 
The University of Minnesota-Duluth, Ohio University, North- 
eastern State Technical and Community College (Tennessee), 
The University of El-Paso, University of Cincinnati, Lynchburg 
College (Virginia), Blue Ridge Community College (Virginia), 
Santa Fe Community College (Florida), and Washington State 
University. These institutions participated for a variety of reasons, 
including preparation for accreditation self-studies, implementa- 
tion of new academic information systems, and a desire to focus 
more on assessment. 

The Data Audit and Analysis Toolkit describes a process that 
an institution would likely want to complete once every five or 
so years in order to assess its approach to the first college year 
and develop a complete inventory of current available data. 
It provides a step-by-step explanation of how personnel at an 
institution can inventory both official and unofficial data in use 
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at the institution and can serve as a first step in developing a "culture of data use" 
at an institution. The process described in the Toolkit is a method for identifying 
institutional inefficiencies in terms of data overlaps or gaps. In addition, it is also a 
method for identifying inconsistencies in data definition and use across the institution. 
While the focus of the Toolkit is on the first year of college, colleges and universities 
can expand the process beyond the first year. 

Highlights of data audit procedures are excerpted from The Data Audit and Anal- 
ysis Toolkit below. The Toolkit provides a broader and more advanced step-by-step 
process for conducting a data audit. Please note, however, that although institutions 
should strive to accomplish these steps, it is important to be flexible in carrying 
out the task. 

1. Identify offices and units across campus that gather or keep data as well 
as those offices and units that use or report data. 

2. Contact appropriate individuals who can fairly represent the resources 
and perspectives of these offices and units. 

3. Set up mutually agreeable times to visit these individuals in their offices 
in order to discuss data sources and data uses. 

4. Approximately one week before the visit, send these individuals a list 
of the questions to be discussed and the artifacts or documents you will 
want to collect from them. If a particular office is only a data-source 
office or only a data-use unit, adjust the list of questions accordingly. 

5. Conduct the site visit. Ask your questions. Clarify, clarify, clarify. Take 
detailed notes. Collect artifacts and documents. Where appropriate, 

"walk through the process" by simulating the steps a student (or 
faculty /staff member) would take, or follow the path of a particular 
data element from point of collection through data entry, archiving, and 
use. 

6. Before leaving, thank the people involved for their time and help. Invite 
them to contact you if they think of anything further that might be of 
use. Secure an agreement that should there be any follow-up questions, 
they will be willing to respond to them. Confirm their telephone 
numbers or e-mail addresses. 

7. Send thank-you notes to people you visited and interviewed. 



Different institutions may require somewhat different approaches because of 
their organizational structures and politics. At the same time, an office or individual 
(usually the Office of Institutional Research or its equivalent) may have already ac- 
complished much of the work included in a data audit. Where this is the case, it is 
useful to refer to this earlier work as a starting point. A previous data audit does 
not preclude a second one as things may have changed or been overlooked in the 
earlier process. 

Based on feedback from participants at the pilot institutions, the Toolkit includes: 



Introducing The Data Audit and Analysis Toolkit 5 



• ATechnical Manual for institutional personnel who will be conducting day-to-day 
activities, including interviews that are part of the data audit. Participants in the 
pilot study pointed out the importance of having the flexibility to copy pages as 
needed as well as to interleave pages of local institutional information with Tool- 
kit pages. Thus, the Technical Manual is published in a three-ring binder format. 



• An Administrative Rationale offering an executive overview for administra- 
tors. While administrators are key to creating a culture of data use on campus, 
they do not need to understand the process of conducting a data audit in detail. 
The Administrative Rationale contains an abbreviated version of the contents 
of the technical manual with just the chapters about data use. This shorter 
pamphlet can be given to institutional administrators to apprise them of what 
a data audit entails and how it can be a launching point for a culture of evidence. 



• A CD-ROM containing the data element lists that encompass data often 
included in student information systems, assessment offices, personnel sys- 
tems, service office units, and facilities. Similarly, examples of charts and ta- 
bles that have been found useful when looking at first-year activities at other 
institutions are also included in both hard copy and electronic forms. How an 
institution uses these resources depends on its needs. One institution might 
already collect all the data elements; another might find data it would like to 
start collecting. Institutions can use these lists as guides to determine if they 
are gathering data they want to in order to conduct more thorough analyses 
of the first college year. 



The Toolkit is not a method for collecting new assessment data. It is not a plug-in 
electronic tool for calculating analyses. It does not contain new survey instruments, and 
it is not an ongoing expense. Rather, The Data Audit and Analysis Toolkit is designed to 
support the early steps an institution takes to determine what data are available on cam- 
pus about first-year students. The audit allows an institution to identify existing data col- 
lection routines as well as what data are necessary for reporting to various groups, both 
internal and external. Once data are known, better analyses and better uses of data can 
be achieved. Even at institutions where few data are collected, an understanding of what 
data are available and how it is used can lead to more nuanced and targeted discussions 
about assessment of the first year of college. 
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First-year program administrators are challenged with 
many tasks — creating, managing, assessing, and re-evaluating 
programs. With so much to do, it is important to consider how 
FYE administrators can most efficiently manage their duties. 
One way to be efficient is to know what information about the 
institution and its members is already available. Happily, the 
institution's Office of Institutional Research (IR) has data and 
information that can be of great assistance. There are several 
important reasons why the IR office should be a first stop for 
information. Because of the information collected and reported, 
IR officials are very knowledgeable about students. In addition, 
they are knowledgeable about assessment techniques, and trends 
in the numbers of students, faculty and staff, and they often 
serve as campus consultants for the development and analysis 
of surveys and focus groups. 

Because the mission of a traditional IR office is to gather, 
store, report, and analyze data, IR officials may likely have data 
that is important to FYE administrators. This data may exist in 
both standard reports of raw data, as well as written reports that 
summarize information. For example, IR professionals gather 
and maintain relevant information about faculty and staff to per- 
form benchmark and peer studies. IR officials may also develop 
written reports that compare faculty and student perceptions 
or white paper-type reports that examine current trends in in- 
structional methodology, faculty satisfaction and workload, why 
students drop out, career plans of graduates, or the confluence 
between in-class and out-of-class learning. Some IR officials may 
also serve as campus assessment experts and can be helpful in 
sharing information on trends in instructional approaches or how 
faculty and students are managing the combination of technol- 
ogy in the classroom. Since IR professionals are knowledgeable 
about the campus, its students, faculty and staff, as well as trends 
in higher education, they are invaluable resources. 

IR officials obtain information through two major collection 
efforts: (a) information retrieved from campus databases and (b) 
information collected through new efforts including Internet data 
searches, surveys, and focus groups. 



Information Obtained Through Computer Databases 



IR officials regularly collect information about students, 
staff, and the campus that is used for state and federal reports 
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(e.g.. Integrated Postsecondary Educational Data and Statistics (IPEDS), state system re- 
ports), accreditation or major survey reports such as US News & World Report, as well as for 
internal management. For example, IR colleagues likely create reports on student gradua- 
tion and retention rates, transfer patterns into and across the institution, and the percentage 
of students who change majors. In addition, IR officials collect and report information 
on campus demographics including the number of students, faculty, and staff; gender; 
ethnicity; ability measures (e.g., SAT and ACT scores); number of students who live on 
versus off-campus; student participation in Greek, honors, and other social activities; and 
the number of students who transfer into or out of the institution. Some IR offices even 
collect and report on other campus information that may be helpful such as level of faculty 
participation in and out of the classroom, room utilization, and the percentage of faculty 
and students who participate in undergraduate research or other campus programs. 

Information Obtained Through New Data Collection Efforts 

Often times, college and university officials need data that are not already available. 
When this happens, IR officials may initiate a study or other data collection effort in order 
to retrieve that needed information. New data may be collected through unique (one-time 
only) data acquisition from peer institutions but is most often collected through surveys and 
focus groups. 

Data Retrieved from the Internet 

The Internet provides access to much information that was previously acquired by mail 
or telephone request. If a specific question needs to be asked quickly, IR officials may do a 
search on the Web, or go to a peer institution's home page. Many institutions include a link 
to their common data set (most often from the IR home page or through the Common Data- 
set Exchange web site), a robust set of information about the institution and its members. In 
addition, Internet sites such as the National Center for Educational Statistics Peer Analysis 
System or WebCaspar enable IR professionals to search quickly for specific higher education 
information. Within these datasets, there is a wealth of information that first-year program 
officials may find helpful. For example, the Peer Analysis System offers information on the 
size and make-up of each institution's student and staff population, tuition and fees, degrees 
awarded, and financial picture. In addition to standard data retrieved about students, the 
Internet enables the researcher to review scholarly information published in books, journals, 
and e-zines. Searches through electronic journals can be a great time-saver when looking for 
the latest research on college students. 

Surveys 

A majority of new information collected by IR officials is accomplished through survey 
administration. Whether the information is gathered through a paper /pencil, web-based, or 
telephone survey, IR officials often gather information on levels of faculty and student satisfac- 
tion, perceived needs or growth of students, opinions on the quality and/ or use of campus 
services, and perceptions about student learning. There are a variety of published instruments 
that focus specifically on first-year experiences of students. These instruments include those 
available through the Policy Center on the First Year of College, Your First College Year Survey, 
The National Survey of Student Engagement, The Student Opinion Survey or Student Needs 
Survey, The Student Satisfaction Survey, and the Admitted Student Questionnaire. 
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Focus Groups 

Often, IR officials will use focus groups to gather qualitative information that may 
serve as complimentary information to quantitative surveys. While focus groups are 
time-consuming and require facilitators who are knowledgeable in this technique, they 
can provide a wealth of information about students and their perceptions of the campus. 
For example, a series of focus groups may be held with first-year students during or im- 
mediately after they have completed a first-year experience program. Results can yield 
helpful information on the benefits and challenges of a FYE program as well as suggestions 
for improvements in the future. 

The Fact Book and Other Written Reports 

From the variety of information collected in the IR office, a complement of written 
reports may be available. Some of the information collected in the IR office is compiled in a 
publication called a Fact Book, which contains much helpful information about the numbers 
of students and staff; enrollment, graduation, and retention patterns; library resources; 
facilities; and budget information for your campus. 

The institution's IR office has a wealth of data that can be helpful to FYE administra- 
tors. As faculty and staff begin to evaluate their FYE program, the IR office's web site or 
IR colleagues can provide information and resources regarding activities to consider for 
their program. 

Web Resources 

NCES (National Center for Educational Statistics) Peer Analysis System 

http: / / nces.ed.gov / ipedspas / 

WebCaspar 

http: / / caspar.nsf.gov / cgi-bin / WebIC.exe?template=nsf / srs / webcasp / start, wi 

Policy Center on the First Year of College 

www.brevard.edu/ fyc/ resources/ index.htm 



Using EnrollmentSearch to Track First-Year Success 

One-quarter of all first-year students do not return to 
the same college for their sophomore year (ACT, 2003). Less 
than half obtain a degree from their first institution (American 
Council on Education, Center for Policy Analysis, 2003). These 
statistics reflect the difficulty that a large number of first-year 
students face in adjusting to college life and its academic de- 
mands, leading many to transfer to different institutions or 
drop out of college altogether. 

For years, colleges and universities have grappled with 
the problem of how to improve first-year to sophomore reten- 
tion, developing different types of programs and strategies 
to increase the likelihood that first-year students will return 
for a second year and, ultimately, graduate. It is important 
for higher education and community leaders to understand 
how well their initiatives are succeeding because the impact 
of low retention can be costly, both to institutions and their 
students. 

As most college administrators know, it is substantially 
more expensive to recruit new students than retain existing 
students. In addition, while successful students are likely to 
recommend their colleges to others and become enthusiastic 
alumni supporters, the negative word-of-mouth from those 
who have had unsatisfactory experiences can be very damag- 
ing. Students also pay a high price for unsuccessful first years, 
including the possibility of never completing their education 
or failing to obtain well-paying and fulfilling jobs. 

Unfortunately, it can be difficult for institutions and 
communities to understand how to focus resources or mea- 
sure the effectiveness of existing programs if they do not 
understand the scope of the problem. How many first-year 
students leave and complete their education elsewhere? Which 
student groups are most at risk of not returning? How suc- 
cessful have existing first-year intervention and college pre- 
paredness programs been in increasing retention rates and 
student success? 

Answering these questions requires access to enrollment 
information that allows institutions to accurately trace the 
educational path taken by former students across state lines 
and over time. Many institutions have turned to Enrollment- 
Search, an education research service from the National Stu- 
dent Clearinghouse, to obtain the comprehensive and reliable 
student enrollment data they need. 
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What is EnrollmentSearch? 

EnrollmentSearch is a subscription-based service that provides access to the Clearing- 
house's 60 million student records, the nation's largest database of enrollment and degree 
data. Through EnrollmentSearch, institutional researchers and college administrators can 
obtain factual information on the college attendance, persistence, and degree attainments 
of their prospective, current, and former students. 

EnrollmentSearch improves upon traditional methods of data collection, such as stu- 
dent interviews and surveys, which often produce unsatisfactory results. Student interviews, 
for example, can only reveal students' intention to enroll, not confirm whether or not they 
did enroll (or where and when). Survey responses are usually low (just 10-30%) and can 
take a long time to obtain and compile, resulting in incomplete and stale information. Re- 
sults of large volume EnrollmentSearch inquiries, however, are based on actual enrollment 
records and can be delivered in as little as one week. 

EnrollmentSearch subscribers can query the Clearinghouse's database to answer key 
questions about first-year students who have left their institution, including: 

• Which students dropped out of college and which transferred to another 
institution? 

• Where did transfer-out students go? 

• What was their enrollment status at their new institution (e.g., full-time, 
part-time)? 

• Did transfer-out students graduate? 

• What degrees and majors did former students pursue? 

Uncovering the subsequent educational experiences of drop-outs can help institutions 
understand how well they or individual programs are preparing students to complete their 
education, whether at their own or other institutions. 

What is in the National Student Clearinghouse Database? 

Nearly 2,700 post-secondary institutions, representing 91% of the currently enrolled 
students in the United States, regularly update the Clearinghouse database with reports 
on all of their enrolled students, including whether or not each student has graduated. 
Institutions that participate in the Clearinghouse's free degree verification service, Degree- 
Verify, (representing 40% of the nation's degree conferrals) also provide detailed degree 
information (e.g., title, major). 

The amount of detailed degree information contained in the Clearinghouse database 
is growing rapidly as the number of schools participating in Degree Verify increases. The 
Clearinghouse expects to provide accessibility to half of all degrees ever issued by the 
summer of 2004. The Clearinghouse operates EnrollmentSearch in full compliance with 
the Family Educational Rights and Privacy Act (FERPA), which protects students' privacy 
rights in their education records. 

How Institutions Are Using EnrollmentSearch 

One example of how institutions are using EnrollmentSearch to better track first-year stu- 
dent success can be found in a research study conducted by Prince George's Community 
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College (PGCC) (Boughan, 2003). PGCC, located in Largo, Maryland, turned to Enrollment- 
Search to augment data it received from the state, which only covered community college 
student movement to four-year public schools within Maryland. 

Using EnrollmentSearch, PGCC found that its transfer-out rate for the cohort it was 
studying, 1996 first-time first-year students, was nearly three times as large as when it used 
the in-state data alone. The following are excerpts from the study: 

Augmenting college graduation data with Clearinghouse data made it possible 
to develop a much more realistic (and impressive) portrayal of the true extent 
of student success at PGCC. Thirty percent of Cohort 1996 students received an 
academic award, a transfer, or both an award and transfer within six years of 
starting their studies at PGCC. The largest category of this successful student 
group transferred to a senior college or university without first earning 
a degree. . . 

But having comprehensive transfer data is also a great help for conducting 
the flip side of academic success research: analyzing student dropout. It 
prevents untracked transfer students from being included in the sub-cohort 
of students leaving college without apparent academic achievement. This 
allows for an accurate assessment of the extent of dropout, and opens up the 
possibility of properly identifying the causes of negative student outflow. . . 

The firm identification of a significant "successful dropout" population in 
PGCC's student body (18% of the whole cohort) has major implications for 
college retention programs (Boughan, 2003, p. 3). 

Brigham Young University (BYU) also employed EnrollmentSearch as an alternative 
to the data source it usually used, surveys, for a recent graduation research report. Previ- 
ously, BYU's Office of Institutional Assessment and Analysis had mailed out as many as 
30,000 surveys over a period of several years to collect data for similar studies, but found 
the information was often incomplete and the samples they collected too small. 

Among the three cohort groups that BYU studied were "Entering Freshmen," first- 
year students who began at BYU during the 1992-1996 summer terms or fall semesters. 
BYU sent the Clearinghouse 30,852 records. The Clearinghouse used this information to 
search its database and identified the students' enrollment statuses. With the addition of 
Clearinghouse data, BYU determined that, on average, 73% of the entering first-year stu- 
dents graduated from BYU and 4% transferred and graduated from another institution. In 
addition to undergraduate completion rates. Clearinghouse data was useful in discover- 
ing that 9% of the 1992 cohort had also completed a graduate degree (The Clearinghouse 
Record, 2003). 1 



How Does EnrollmentSearch Work? 

EnrollmentSearch subscribers are provided with secure web access to the Clearing- 
house database, including current and historical enrollment records, in order to perform 
individual student queries. Users can search the database to identify or confirm a specific 
student's enrollment at other institutions before, during, and subsequent to the student's 
attendance at their own school. Data for large-scale analyses can be obtained by submitting 
an electronic file of the cohorts being studied (e.g., first-year students who participated in 
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an intervention program and those who did not) to the Clearinghouse. After searching its 
nationwide database, the Clearinghouse will return an electronic list of each previously 
enrolled student to the subscriber. 

The EnrollmentSearch file contains records of each former student's enrollment and 
graduation achievements at postsecondary institutions across the country, including: 

• Name of institution 

• Type of institution (four-year and above, two-year, less than two-year) 

• Attendance dates 

• Enrollment status (full-time, half-time, less than half-time, graduated) 

• Degrees earned and major courses of study (provided by DegreeVerify 
participants, representing 40% of degrees awarded in the U.S.) 2 

A summary report of the enrollment and degree outcomes for the students being 
studied, including totals by college or university, is provided along with the student-level 
detail file. Enrollment records going back to the mid- to late-1990s are available for most 
institutions, allowing users to perform historical studies or develop a longitudinal database 
to study patterns over time. 

Subscribing to EnrollmentSearch 

Any post-secondary institution is eligible to participate in EnrollmentSearch, provided 
it is a participant in the Clearinghouse's student loan verification service. Education agencies 
and organizations are also eligible to participate to support studies for the improvement 
of education. 

EnrollmentSearch subscribers pay a small annual fee, which covers all inquiries made 
during a 12-month period. During that time, EnrollmentSearch can be accessed as often as 
needed. Nearly 1,000 colleges and universities already use EnrollmentSearch to perform 
institutional research. 

Notes 

'To read more about the PGCC and BYU studies as well as view other EnrollmentSearch 
case studies, visit http: / / www.studentclearinghouse.org / colleges / esearch / case_studies. 
htm. 

2 Institutions must be DegreeVerify participants to access degree records. 
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Using Archived Course Records for First-Year Program Assessment 



Archiving historical records of courses is an assessment 
method that is based on the tenets of action research — promot- 
ing educational reform from within by using self-examination 
and focused inquiry. Such an approach assumes that teaching 
is a dynamic process that is based on documenting changes and 
reflecting on how well these changes do or do not accomplish 
agreed-upon objectives. This is especially important in assess- 
ment of first-year courses. First-year or introductory courses are 
rarely "owned" by a specific professor or group of professors, 
which makes continuity and fidelity to course content especially 
important. An analysis of how courses and programs change 
over time may also help interested researchers track and explain 
large-scale paradigm shifts within a given discipline. On a prac- 
tical level, this work is important because it provides a way to 
understand how the content and processes of education change 
over time, so that these changes can be evaluated and ultimately 
influenced by purposeful choice, not unwittingly motivated by 
influences irrelevant to education itself, such as preferences of 
the instructor that are unrelated to course objectives. As applied 
to evaluation of first-year program courses and the students who 
enroll in them, achieving the right balance of change and continu- 
ity in a course may be a significant factor in student retention. 

How can a program ensure that the courses offered, espe- 
cially courses taught to first-year students, maintain fidelity to the 
content and objectives, while allowing for relevant improvements 
over time? Academic institutions with a significant interest in 
accountability for course content may compile archived course 
records, or "course diaries" for use by faculty and administrators, 
especially those preparing for the next offering of a course. A 
course diary documents the content of a course including assign- 
ments, texts used, subjects discussed, handouts distributed, and 
any other pertinent information and supplies a record of overall 
student performance and expectations. Historical archives of 
course records also help faculty members guard against "content 
creep" — the addition of more content with each new offering of 
the course — and "curriculum drain" — the elimination of content 
related to the course goals and objectives. 

A course diary consists of several sections. The first section 
should contain the course description, a discussion of how the 
course relates to the overall curriculum, the number of sections, 
the number of students, classification of the students (i.e., be- 
ginning versus advanced), the texts used, the number of book 
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chapters addressed, the frequency of tests and other assessments, grading criteria, and 
final course statistics (e.g., course point averages, percentage averages, grade distribution, 
and student ratings data). The second section contains everything pointing to that which 
occurred during the course, including the syllabus, handouts, a list of videos viewed, simu- 
lations or experiments conducted, course exams, and other assessments (including scoring 
rubrics). The final section contains a summary of observations and recommendations by 
the instructor or students and recommendations for the future as to improvements or gen- 
eral changes. Based on this information, faculty and administrators can evaluate the need 
for changes and the impact of past changes. Some programs have sought to limit course 
changes by specifying an outside percentage of the content that they will tolerate to be 
changed from year to year, to create continuity and limit "content creep" or the converse, 
"curriculum drain." 

The practice of keeping course diaries requires a minimal time commitment and yields 
information which can be used to make decisions about how courses should be improved, 
while highlighting the successful aspects of the course that should not be changed. Course 
diaries provide documentation for an instructor or inquirer to both analyze change and 
explicitly plan for and justify needed change. Course diaries may be especially useful for 
new courses or courses undergoing significant change. 

Beyond the specific advantages of course diaries for archiving course content, the 
use of archival course records represents a larger commitment to inquiry-based teaching. 
Teaching has too often been identified with informal interactions between teachers and 
students in a classroom, an art form beyond the scope of critical analysis. While teaching 
is composed of informal interactions, it can also, like other forms of scholarship, be seen as 
a process that embodies vision, design, interactions, outcomes, and analysis as it unfolds 
over time. It should be an activity which prompts critical questions about the scholarship 
of the course's content, the effectiveness of the instructional design, the assessment of stu- 
dent learning, and the use of student feedback during the course. Course diaries provide 
the documentation to assist in this important process of critical analysis of change in the 
content and processes of education. 

First-year programs that have used course diaries report greater control over student 
outcomes, increased student retention, and more involvement from courses typically 
relegated to low-priority status. This means that faculty who might otherwise be happy 
to pass off a first-year seminar to a junior faculty member are more interested in teach- 
ing it again. In addition, faculty have a vehicle for reflecting on the effectiveness of the 
course and their teaching, which empowers them to make necessary changes. Finally, 
departments use course diaries to limit grade inflation, which can occur as an unintended 
consequence of reducing or changing curricular content. By maintaining rigor and con- 
tent of assignments and course content, first-year courses can be an accurate indicator to 
students of the program they represent and can be a powerful vehicle of attracting and 
retaining desirable students. In short, course diaries are a wonderful way to promote 
inquiry-based teaching and to address the challenges of first-year course evaluation. 
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Absences, especially in the first-year population, can iden- 
tify students who may be having some difficulty adjusting to the 
responsibilities of being a university student. The University of 
Mississippi developed an initiative called Freshmen Absence- 
Based Intervention (FABI) to monitor and study the relationship 
between classroom absences and student success. The program 
evolved out of research conducted during the 1999-2000 and 2000- 
2001 academic years that focused on the retention of students at 
the University, particularly first-year students. Coordinated by 
the Academic Support Center (ASC) and combining the efforts 
of ASC staff, graduate student clinical research assistants in the 
Department of Psychology, faculty and staff in the Department of 
English, and staff in the Chancellor's Office, the research revealed 
a direct correlation between student absences and student grade 
point average (GPA). 

Based on data from the study and information obtained 
from student, faculty, administrator, and staff focus groups, an 
Absence-Based Intervention Project was introduced and piloted 
during the 2000-2001 academic year in the Department of English. 
The program, which initially involved 589 students, focused on 
reporting classroom absences and implementing subsequent in- 
terventions. One goal of the project was to establish a possible 
relationship between absences and success in college, specifically 
the correlation between absences and GPA. 

Faculty involved in the project reported excessive absences 
at specified times during the semester via an electronic form de- 
signed by Information Technology. Students enrolled in English 
101 courses were divided into a control and intervention group. 
In the intervention group, students with two or more absences 
received personal contact or "an intervention" via telephone 
or visit from a graduate student. They were given information 
about the support services available at the University. For the 
control group, faculty reported absences, but no intervention 
was conducted. 

Data from this pilot program revealed a direct correlation 
between absences and success at the University. Further, students 
who received intervention attained significantly higher GPAs 
compared to the students who did not receive intervention. Out 
of the total sample of students in the experiment group, 39% re- 
ceived telephone intervention. In addition, 41% of the students 
who received telephone intervention had two or more absences 
after only three weeks of classes. 
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Course performance indicated that early intervention was crucial to future success. 
For example, data revealed that more students in the intervention group than in the control 
group earned midterm and final grades of a "C" or better. Specifically, 87.4% of students in 
the intervention group obtained a "C" or better at midterm, whereas only 62% of students 
in the control group attained such grades. Final grades revealed a more drastic difference: 
87.4% of the intervention group ended the semester with a "C" or better, whereas only 55% 
of those in the control group did. 

Based on the success of the pilot program, FABI was introduced to the campus com- 
munity at large in the fall of 2001. With campus implementation, certain particulars of the 
pilot study were changed so that program implementation could be achieved on a much 
larger scale. For example, counselors in the University Counseling Center suggested that 
peer groups might serve as the best vehicle to initiate intervention. As a result, the actual 
delivery of the intervention was conducted in person by the student's resident hall advisor, 
as opposed to telephone intervention. 

Because the program depended on faculty support and cooperation, faculty had to 
be brought on board. At the start of the fall semester, faculty received a letter from the 
Provost's office introducing FABI as a program coordinated by the Academic Support Cen- 
ter that targeted first-year student attendance in lower -division courses. Faculty learned 
that the program was not designed to evaluate attendance policies; rather, it stressed the 
importance of class attendance. Excessive absences may also contribute to poor academic 
performance. Staff in the ASC asked faculty who taught lower-division courses to report 
on a secure web site first-year students with excessive absences. Instructors determined 
what was excessive for their courses. 

The FABI coordinator downloaded the information provided by faculty once a 
week beginning the second week of classes and continued the download for eight 
weeks. The program ended mid-semester since determination of a student's standing 
in a course was determined at that time by a mid-semester grade. After student names 
were downloaded and carefully checked for continued class enrollment, resident hall 
advisors (RHAs) received a list of students via e-mail for one-on-one intervention. The 
RHA gave the student an informational packet provided by the ASC that stressed the 
importance of class attendance. RHAs were asked to report back to ASC staff as soon 
as contact was made. Neither the student nor the RHA knew which instructor reported 
the absences. They only knew that an instructor reported that the student had missed 
classes. The RHA contacted a student one time only. Any additional intervention, if 
necessary, was made directly through ASC staff either through e-mail or by telephone. 
Similar procedures were carried out during subsequent semesters. In addition to letting 
students know that the University was indeed paying attention to them, the contact made 
with students provided advisors in the ASC an opportunity to discuss various support 
units/ services available on campus. 

In Fall 2001, approximately 41 instructors of first-year courses reported students with 
excessive absences, and 245 students were contacted by RHAs concerning excessive ab- 
sences. Among students receiving a letter grade in a course, 58% of students reported for 
absences and receiving RHA intervention earned a passing grade, while 42% of students 
reported for absences and receiving subsequent intervention failed the course. 

In Spring 2002, approximately 53 instructors of first-year courses reported students 
with excessive absences. Approximately 484 students were contacted concerning excessive 
absences through RHA notification. Among students receiving a letter grade, 70% of stu- 
dents reported for absences and receiving subsequent intervention earned a passing grade, 
while 30% of students reported for absences and receiving intervention failed the course. 
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In its first year, FABI demonstrated that students with excessive absences were at risk 
when it came to class grade point average. Although the pilot study (with control and ex- 
perimental groups) suggested that such an intervention program does influence attendance 
and academic performance, there is not yet enough information to determine whether the 
program directly affects classroom success when practiced campus-wide. One ancillary 
benefit of the program is that it promoted open discussion of the importance of classroom 
attendance and University intervention concerning the actions of its first-year students. 

Administrators, faculty, and staff at The University of Mississippi have demonstrated 
through their stated goals and initiatives the commitment to attract, retain, and graduate 
its students. For an institution striving to enhance its national profile and reputation for 
high-quality programs and academic excellence, initiatives devoted to helping students 
admitted under liberal admissions policies to meet higher expectations are crucial. While 
the first-to-sophomore retention rate at The University of Mississippi is about 76%, a good 
rate for institutions in our ACT admissions category, the chancellor and provost have set a 
goal of 80% first-year-to-sophomore retention. We firmly believe the ongoing work of the 
Freshmen Absence-Based Intervention program will help us attain our goal. 
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Focus group research is a form of qualitative methodology 
used to gather rich, descriptive data in a small-group format from 
participants who have agreed to "focus" on a topic of mutual in- 
terest. Krueger and Casey (2000), in their comprehensive book on 
focus groups, characterize a focus group as 



. . .a carefully planned series of discussions designed 
to obtain perceptions on a defined area of interest in 
a permissive, non-threatening environment. Each 
group is conducted with six to eight people by a 
skilled interviewer. The discussions are relaxed, 
and often participants enjoy sharing their ideas and 
perceptions. Group members influence each other by 
responding to ideas and comments of others, (p. 5) 

Focus group methodology provides insight into a topic from 
the participant's point of view. Focus group research addresses the 
"why" of issues. In contrast, quantitative methodology emphasizes 
collecting numerical data, frequently answering the question of 
"how many" In focus groups, participants are encouraged to speak 
in their own language and to move the topic-at-hand to the most 
meaningful points for discussion. Emphasis is on understanding 
of participants' experiences, interests, attitudes, perspectives, and 
assumptions. 

Focus groups are a valuable methodology for investigating 
programs, services, problems, and products in higher education. 
They can be used to generate new ideas, explore an issue in-depth, 
and understand critical issues in planning and delivery of pro- 
grams and services (Jacobi, 1991). Although focus groups may be 
used alone, they are increasingly used in combination with other 
methodologies such as interviews, observations, or surveys. Meth- 
odological triangulation is the use of multiple methods to study 
a single problem (Denzin & Lincoln, 2000). As a complementary 
methodology, focus groups may be used to interpret puzzling re- 
sults from surveys, to guide the development of a questionnaire, or 
to give voice to quantitative results. In triangulating methodologies, 
a broader perspective is gained on important research findings and 
the likelihood of misinterpretation of results or outcomes may be 
decreased. Increasingly, qualitative researchers use the term "crys- 
tallization" instead of triangulation to represent the use of multiple 
lenses and perspectives in multi-faceted research designs. 

Focus groups may be useful for gathering data from specific 
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populations on a wide variety of topics. Examples of focus groups described in the litera- 
ture include college-bound high school students giving feedback on recruitment materials 
(Benson, 1999), transfer students focusing on the transfer process (Davies & Dickman, 1998), 
African-American students sharing perceptions of being a student on a traditionally white 
campus (Lee, 1999), and distance education students focusing on the quality of distance 
education courses (Purnell, Cuskelly, & Danaher, 1996). 

Focus Group Cycle: Planning-Conducting-Reporting 

Three distinct stages of the research cycle — planning the focus group research, con- 
ducting the research, and analyzing and reporting results — are discussed below. 

Planning for the Focus Group 

Several steps are important to ensure an effective focus group. First the sponsoring 
office or institution (hereafter called the planner) must be clear about the goals of the 
research. A great deal of discussion should surround the question "What do we want to 
know?" It is important to outline the problem, issue, or product to be assessed; to identify 
the purpose of the focus group; and to generate a list of the broad research questions at the 
earliest stage of planning (Ponsford & Masters, 1998). 

Subsequently, the planner must decide on the composition of the focus group. The 
composition of the group is guided by the fundamental question "Who can most clearly 
answer these important research questions?" The planner should identify the ideal par- 
ticipant and the criteria for participation. 

Students are most often the participants in college and university focus groups; yet, 
effective groups may include employers, alumni, staff, or any other stakeholders involved 
in the process, programs, and outcomes of higher education. Again, the research questions 
guide the selection of focus group participants. Overall, you want to attract people who 
will give valuable feedback, for whom the topic is important, and whose responses will 
not be biased by their assumptions of what you want to hear. 

Generally, groups that are homogeneous perform better than heterogeneous groups. 
Thus, in planning the number of groups and group composition, the potential impact of 
gender, race, affiliation (e.g., Greeks or non-Greeks), age, experience, and other attributes 
on the performance of the group should be considered. Research shows that significant 
differences within a group may render individuals who are unwilling to talk and interact 
in the group; consequently, it may be necessary to convene more than one group and vary 
the composition by group. For example, focus groups on library services might convene 
one or more groups of traditional undergraduate students, other groups of non-traditional, 
older students, and a third set of distance education students. By increasing the number 
of focus groups and by varying the composition of each, the depth and variety of findings 
may be maximized. 

The population identified as the subjects for the research make up the "sampling 
frame" — that population of potential subjects from which the researcher will select focus 
group participants (Bers, 1990). For example, a study of student attitudes toward service- 
learning might include in the sampling frame all students who did service-learning in the 
previous semester. A second cut might include only those who did service-learning in their 
major. A third cut might include only those who participated in semester-long versus one- 
time service-learning activities. 

Most important in creating the sampling frame is the experience of the potential 
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participant with the activity under focus. For example, if the focus group will be used to 
investigate experiences in the first-year seminar, all students who enrolled in the seminar 
will initially be in the sampling frame (i.e., those students who have experience in the 
activity). Additional criteria for participation could include (a) only those students who 
completed the seminar and (b) students with no direct leadership role with the program. 
Based on these criteria, screening questions might read: "Did you enroll in a first-year 
seminar?", "Did you complete the seminar?", "Have you been a peer leader in the first-year 
program?" Given otherwise eligible participants, several groups then might be formed based 
on gender, residential status, or other attributes as deemed appropriate. Clear criteria for 
participation is necessary to develop screening questions that include certain participants 
while excluding others. 

Research questions define the sampling frame while specific criteria for participation 
leads to the creation of screening questions used in selecting specific participants. "Com- 
monality" in the experience is often the most important criteria for participation. This is 
not to be confused with sameness of opinion or attitude toward the issue under discussion. 
Clearly, focus group participants who know nothing about the topic under discussion add 
little to the research. 

Following the development of the "screen" (i.e., the questions used to screen the 
potential participants), planners must solicit participants for the focus group. Screening 
may take place by telephone or e-mail (or in other creative ways) using a list of potential 
participants. Participation may be solicited through an announcement to a listserv, a visit 
to a classroom, or through a posting at places where students gather. I will not discuss 
the pros and cons of convenience samples versus more systematic sampling here, but I 
will advance that students who meet the criteria for your study will likely generate useful 
information for analysis. 

Not to be forgotten in the planning phase is making clear arrangements for where 
and when the group will meet and for communicating this to the invited participants. Re- 
minding participants of the scheduled focus group is paramount, especially the day before 
the planned activity. Focus groups fail if no one comes! Also to be decided in advance is 
whether incentives will be used to encourage participation in the focus group. A small in- 
centive may increase the inclination to participate. In the commercial use of focus groups, 
participants are paid; this also happens in higher education. With students, the more likely 
scenario is food at the event or coupons for later use. 

Another important planning issue is the role of the moderator in planning, conduct- 
ing, and evaluating the focus group activity. Moderators may come in at any stage, but 
this decision needs to be made early in the planning cycle. Although much is written about 
the role of moderators and the how-to of conducting the group, it is most important that 
the moderator be a good listener and discussion leader, not identified with any particular 
outcome, and have no direct authority over the participants (e.g., program director or in- 
structor). The moderator should be seen by the participants as an interested, but neutral 
party to the outcomes of this study. 

Conducting the Focus Group 

On the day the focus group will meet, it is important to arrive early, organize the room, 
and set-up the audio or videotaping equipment (participants must grant their permission 
to be audio or video-taped). When the participants arrive, someone should greet them and 
make each feel comfortable and valuable to the process. The participants should be screened 
again at this point to be certain they meet the criteria for participation. Extra participants 
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who were invited to ensure a sufficient group size should be thanked, given the promised 
incentive (e.g., food, movie coupons), and dismissed from participation. During the group, 
a logistics person is needed to handle the equipment, take notes, and if agreeable with the 
moderator, to interject a question for clarification or other purposes. This person may be 
the same as the greeter. Finally, the moderator will begin and end the group session, which 
on average takes 90 minutes or less. 

The moderator should be skilled at using various types of questions, nonverbal be- 
haviors, and non-judgmental cues to elicit responses. The moderator must also be skilled 
at listening, guiding, and prompting. The focus group responses will be no better than the 
skill of the moderator in guiding the discussion and improvising as the group dynamics 
dictate. Moderators should prepare a set of questions prior to the session in consultation 
with the planners for use in guiding, but not controlling, the discussion. The moderator's 
skill at eliciting participation, honest opinions, and managing group process will greatly 
affect the quality of the research outcomes (Greenbaum, 1998). 

Deciding on the number of focus groups is always a challenge. During the course of 
the research, it may be necessary to plan a third or fourth group. Or, it may be necessary to 
cancel a group. The analysis of the data throughout the process will yield clues as to when 
to add groups and when to wrap up the process. When you begin to hear the same themes 
and get the same outcomes, you have saturated the opinions of the target population, and 
data collection may end. 

Analyzing and Reporting the Results of the Focus Group 

Focus group reports may vary in length, style, and comprehensiveness based on the 
needs of the planners. For example, in raw data summaries, the transcript of the session is 
supplied to the sponsors with all questions and verbatim comments. In descriptive reports, 
themes are identified and supported by selected quotes. In interpretive reports, a summary 
of the discussion along with illustrative quotes is followed by interpretation of the meaning 
of the dialogue, questions, and themes. If the moderator is not writing the report, he or she 
may be asked to develop a statement of moderator observations by topics discussed. 

A standard report generally has the following sections. The report opens with the 
purpose of the study: research questions, specific objectives of the research, logistics, and 
sponsors and planners of the research. Next, the group composition is described (preserving 
anonymity for participants). Included is an overview of the selection criteria, the screening 
guide and process, number of participants, and their general characteristics. Another section 
describes the environment in which the focus group was conducted; the way participants 
were assembled and welcomed; the incentives, if any, and their use. 

The results of the focus group constitute the major section of the report. What did 
the focus groups reveal about the research questions? What themes and ideas emerged? 
What important, and unexpected, issues emerged? Were there similarities or differences 
of opinions and perceptions? This section may also include paraphrased or verbatim 
participant comments. 

Following the summary, a synopsis of any problems that occurred should be noted. 
Were there extremely biased members? Did some fail to talk? Moderation problems? Re- 
search is never pure and without problems, and the limitations should be addressed in the 
report. Depending on the nature of the report, it may end with conclusions. In all cases, 
the appendices should include screening instruments, moderator guide, and any materials 
used as prompts or props. 
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Conclusion 

Focus groups allow for spontaneity, synergism, and feelings of security (i.e., it is okay to 
advance an unpopular response). Further, focus group responses often "snowball," building 
one on the other, something lost in individual interviews (Fless, 1968). In the focus group, 
every participant does not have to answer each question as in individual interviews, and 
the group process stimulates group discussion and idea generation, often lost in dyads. 
Focus groups seemingly generate excitement among the participants and yield immediate 
information for planners. 

In summary, focus group research may be a valuable tool for exploring the first-year 
experience of students. The first attempt does not have to be perfect. Believing that we 
"learn by doing" is a reason to give focus groups a try, and if at first you don't succeed, 
that's okay, try again! 
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Looking at the First-Year Experience 
Qualitatively and Longitudinally 

j Belcheir Have you ever conducted a quantitative study — developing 

instruments, gathering data, and tracking students longitudi- 
nally — only to discover at the end that you still felt you did not 
have a handle on the issue? I have. When I first came to Boise 
State University, retention was the biggest issue on campus and, 
therefore, one I tackled early on. After an extensive quantitative 
study, however, I still felt that I lacked an understanding of the 
new students' experience at the university. What did we want to 
know? 

The quantitative study had indicated that re-enrollment 
was best predicted by first-term GPA. First-term GPA was best 
predicted by admission index scores and the numerous conver- 
sations held with faculty during the first semester. In addition, 
students who reported more occasions of feeling lost and alone on 
campus during their first semester had lower GPAs, and students 
who used more services were more likely to remain enrolled. 
Taken as a whole, these findings indicated that preparation for 
college and engagement in the campus resulted in higher GPAs 
and retention. Still, an understanding of what was going on 
during that first semester eluded me. 

Clearly, another kind of study was needed, one that an- 
swered different sorts of questions. These questions included: 
What were students' first impressions as they arrived on campus? 
How did they negotiate the admissions, advising, registration, 
and financial aid processes? What were their classroom experi- 
ences their first semester? Were these experiences what they 
expected? What was meant by "conversations with faculty," and 
who was having them? Finally, of course, all these questions led 
to the larger question of how these experiences related to student 
success (as measured by GPA) and retention. These larger and 
more nebulous questions could not be easily answered by giving 
students structured survey instruments. Gaining a "student's 
eye view" could only be accomplished through another form of 
data-gathering, one that was more qualitative in nature. 

How Did We Conduct the Study? 

Data for the study were acquired through three methods. 
First, the students were interviewed weekly during the fall 
semester. The basis for the interviews was a series of ques- 
tions developed prior to the fall semester with input from both 
academic and student affairs offices. Each week, students were 




31 



32 Belcbeir 



asked a different series of questions. These questions started with their arrival on campus 
and ended with a reflection on their first semester. Second, students kept journals where 
they recorded both positive and negative experiences of all aspects of their lives. Finally, a 
group meeting was held at the close of the first semester during which students met each 
other and the research team face-to-face for the first time. This in-depth, long-term relation- 
ship with students allowed for gathering data not generally available through traditional 
survey methods. 

The 25 students selected for the study were deliberately chosen to represent a cross- 
section of Boise State first-year students. Selection criteria included gender, ethnicity, 
age, and whether or not students had chosen a major. In return for their cooperation, 
we offered each student a $50.00 gift certificate from the bookstore, payable when they 
turned in their journals at the end of the semester, and one free academic credit. An Excel 
spreadsheet contained additional information on demographic variables, entering test 
scores, high school GPA, major, financial aid, living arrangements, credits taken, and jobs. 
Students were assigned a research number and name to be used throughout the study to 
help preserve anonymity. 

Conducting a qualitative study is a labor-intensive process. A half-time staff person 
and work-study student provided support for managing the study, especially for weekly 
calls to students. With student permission, phone interviews were recorded so that I could 
conduct spot-checks and retrieve verbatim comments. The data manager recorded the 
"gist" of the interview responses, including direct quotes for particularly applicable com- 
ments. While verbatim transcripts of the interviews would have been ideal, we simply 
did not have the resources to do it. 

The journals provided a more intimate peek into students' lives. Some students wrote 
in them almost daily, while others wrote weekly. Some wrote only sporadically or not at all 
despite frequent prompting during weekly interview sessions. Where possible, all personal 
identifiers were removed from the journals prior to reading. 

The end-of-semester meeting was the least productive part of the study. Despite of- 
ferings of pizza, only about half of the students showed up, and those who did were not 
very interested in talking. Perhaps the timing of the meeting was a factor — finals were 
looming. Also, they may have been "talked out." In retrospect, a different process may 
have produced better results. 

How Did We Analyze the Data? 

There is nothing cut and dried about a qualitative data analysis. Approaches depend 
upon the philosophy of the researcher, the questions posed (if any), and the data. For our 
study, early questions about arrival and classroom experiences were answered by reading 
responses for themes and patterns, forming possible conclusions, and then re-reading to 
verify or modify those conclusions. 

For the larger question of how background and experiences related to retention, the 
data had to be studied in yet another way. First, a sketch of each student was developed. 
Students also were identified as successful or not successful and patterns were sought 
within each sub-group. The "successful" group consisted of 13 students who returned for 
both spring and fall terms and who maintained GPAs above 2.0 for both semesters of their 
first year. The "unsuccessful" group consisted of seven students who did not re-enroll and 
who also had GPAs below 2.0 for at least one semester. A third group of four students was 
labeled "at risk" because, despite the fact that they continued to enroll, during either their 
fall or spring semester they had a GPA below 2.0 indicating they might be in academic 
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jeopardy. A final group consisted of only one student, who had a good GPA both semesters 
but still did not return one year later. 

What Did We Find? 

Students were somewhat surprised to find large lecture hall classes. Though they 
viewed them as a necessary part of college life, they preferred the smaller class sizes 
where they could participate in discussions and had more access to the professor. A 
majority of students in the study did not have a good idea of how they were doing in 
their classes until semester grades were posted. Most wished for more feedback about 
their performance earlier in the semester. These findings had definite implications for 
scheduling and faculty development on our campus. 

We found that students who were successful represented the spectrum of the student 
body. For older successful students, a key to success seemed to be a supportive home 
environment. For younger successful students, getting involved with campus activities, 
seeing the value of an education, and enjoying the pursuit of learning helped. Family 
support for the younger group was less important than for the non-traditional students. 
In fact, extreme family support could sometimes signal that the parent, not the student, 
was committed to college. 

Students who were not successful typically fell into two groups. One group consisted 
of women who had children and were trying to go to school with little support from 
their households. The other group consisted of 18-year-olds who generally appeared 
developmentally unprepared for college. 

Being able to make finer distinctions among "types" of students was a benefit of the 
qualitative approach. For instance, in the qualitative study we found that first-semester 
GPA was a key variable in predicting further enrollment since students who received 
low grades were much less likely to return. Through the qualitative study, however, we 
noted that some students stopped attending classes prior to the end of the semester and 
therefore received poor grades, indicating that GPA was an outcome as much as it was 
a predictor. A similar finding occurred for faculty contact. In the quantitative study, it 
appeared that more faculty contact predicted higher GPAs. We discovered, however, that 
most of the students talking to faculty were those who were already doing well. Those 
who were performing poorly typically were not talking to their professors, because they 
had bad grades and felt they would be viewed as "stupid." 

A qualitative study is specific to a campus's environment and students. Thus, what 
we discovered might not be true for other campuses. For these and other reasons, I highly 
recommend that campuses undertake their own qualitative research studies of first-year 
experiences. The results will undoubtedly be illuminating, providing institutions with a 
study based on the uniqueness of their own student bodies and campus environments. 

Author's Note 

For those interested in further details of the study, please visit our web site, http: / / www2. 
boisestate.edu/iassess/ . Click on "Reports" then view reports 97-04, Lasting First Impres- 
sions: A Qualitative Study of Freshman Arrival on Campus, 97-06; It's Academic: A Qualitative 
Study of Student Classroom Experiences ; and 98-05, Who Stays? Wio Leaves? Results from a 
Qualitative Freshman Study. 
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"Deep understanding" is what teachers want for students. 
But how do we know when it has been achieved? Are certain 
assessments better than others at shedding light on what students 
really know and understand? Few would defend the method 
used by a deaf, English public schools inspector who, after lis- 
tening to student recitations, would rise and declare, "I have not 
been able to hear anything you have said, but I perceive by the 
intelligent look on your faces that you have fully mastered the 
text." This essay describes the authors' experience with ineffec- 
tive student learning assessments and their subsequent employ- 
ment of an effective technique called the think-aloud method. 

Experienced teachers know that popular assessment meth- 
ods conceal as much as they reveal. Papers and exams, for exam- 
ple, offer little help for figuring out why a student has recorded a 
wrong answer or struggled unsuccessfully with an assignment. 
Conventional assessments also run into problems of validity. 
Because they rely on students' ability to articulate themselves in 
formal language, papers and exams tend to conflate understanding 
with fluency. But sometimes, especially with first-year students, 
the tongue-tied harbor deep understandings even though they 
perform poorly. The reverse is true, as well; sometimes articu- 
late students are able to say more than they really understand. 
"The thorniest problem" of assessment, according to Wiggins 
and McTighe (1998), is differentiating between the quality of an 
insight and the quality of how the insight is expressed. 

A helpful tool for grappling with this problem is think-aloud 
protocol assessment. Think alouds are a research tool originally 
developed by cognitive psychologists for the purpose of studying 
how people solve problems. The basic idea behind a think aloud 
is that if a subject can be trained to verbalize his/her thoughts 
while completing a defined task, then the introspections can be 
recorded and analyzed by researchers to determine what cog- 
nitive processes were employed to deal with the problem. In 
fields such as reading comprehension, composition, mathematics, 
chemistry, and history, think alouds have been used to identify 
what constitutes "expert knowledge" as compared to the thinking 
processes of non experts. For first-year assessors, think alouds 
offer a promising method of uncovering what conventional as- 
sessment methods often miss: hidden levels of student insight 
and / or misunderstanding. 

We first used think alouds when assessing a new design 
for a first-year history course. The new design shifted emphasis 
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away from tidy summaries of historical facts and knowledge toward the central ques- 
tions, methods, assumptions, skills, and attitudes that characterize history as a discipline. 
Students completed eight identical assignments in the course, and student learning was 
measured by comparing the students' first and last papers. The results were dishearten- 
ing. It was the rare student who showed steady progress from week to week, and few of 
the final papers were superior to the first ones. On the basis of this evidence, it seemed 
the new course was a failure. 

But course evaluations and self-reports suggested otherwise. Here, students insisted 
they had learned a great deal, a claim that certainly squared nicely with the intelligent 
looks on their faces at the end of the term. Puzzled by the conflicting evidence, we turned 
to think alouds for help. 

Our procedure was as follows. From 60 students in the course, 12 were selected to 
participate in a think-aloud study, representing a cross-section of students in terms of 
gender, grade point average, and major/ nonmajors. For their participation, subjects were 
paid $10 an hour. In week one of the course, we sat down with each student in a room 
equipped with a tape recorder. After training subjects how to verbalize their thoughts, we 
presented them with documents concerning the Battle of the Little Bighorn, a subject most 
knew little about. Then we asked our subjects to think aloud while "making sense" of the 
documents. This was essentially the same task they would perform eight times over the 
length of the course, though in this case their thoughts would not be filtered by the task of 
composing an essay. With the tape recorder running, subjects read through the documents 
aloud, verbalizing any and all thoughts that occurred to them. When subjects fell silent, 
we would prompt them to think aloud, or to elaborate on their thoughts, as they attempted 
to make sense of the historical evidence. 

Our think-aloud sessions lasted anywhere from 40 to 90 minutes. After all 12 ses- 
sions were completed, the tape recordings were transcribed for analysis. Analysis took 
the form of coding each discrete verbalization in the transcript according to the type 
of thinking it exemplified. We were able to identify 15 different types of thinking pro- 
cesses displayed in the think alouds, from the uncategorizable ("it sure is hot in here") 
to comprehension monitoring ("I don't understand that part") to the six types of his- 
torical thinking we were particularly looking for, such as sourcing a document ("I can't 
trust Whittaker; he wasn't there."), asking a historical question ("I wonder what caused 
this battle."), or recognizing limits to knowledge ("I need to see more evidence than this."). 
After coding each think aloud independently, we used a common rubric to rate each 
subject's proficiency on the six thinking skills taught in the course. For this, we used a 
five-point Likert scale where "1" indicated the undeveloped ability of an average high 
school senior and "5" indicated a sophistication comparable to that of a professional 
historian. We then compared our coded transcripts until reaching consensus on how to 
rate the students' abilities in the six key areas. To prevent our bias as course designers 
from influencing the results, we contracted with an outside analyst to help us code the 
transcripts and rate students' abilities. 

At the end of the term, the 12 subjects completed a second think aloud. When these 
sessions had been transcribed and coded and the subjects' abilities rated, we compared 
the first and second think alouds to determine whether students had made gains in their 
understanding of what it means to "think historically." 

The think alouds opened a fascinating window into the thought patterns of students 
before and after the course. Overall, the think alouds revealed cognitive enhancements 
that were not as dramatic as claimed in student self-reports, but much greater than indi- 
cated by using comparisons of early and late papers. 
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Other surprises were equally interesting. Under-performing students struggled less 
with historical thinking than with reading itself. Moreover, in the second set of think alouds, 
we noted that some of the best insights and meaning making came from students who, in 
the grade book, were steady "C" performers. For them, deep understandings seemed to 
evaporate when they tried to wrestle their thoughts to paper. This told us that we had work 
to do if we wanted to distinguish between assessing understanding and assessing students' 
ability to communicate their understanding. The real roadblocks to learning historical think- 
ing, we discovered, are poor reading comprehension and prose writing. 

On our campus, the potential of think-aloud protocols has not been lost on other fac- 
ulty. For example, library staff are using think alouds to assess how first-year students find 
information when they go to the library. Information gained from the study will be used 
to help library staff identify misconceptions and teach against common missteps students 
make when doing research. 

Think alouds are not perfect assessment instruments. The advantage of think alouds 
is that they give us insight into our students' struggle to formulate problem-solving strate- 
gies, employ skills, and develop insights. Papers, exams, and ex post facto commentary 
by students are helpful in their own ways. But they make the process of understanding 
seem more orderly than it is, covering up the confusion, the disorientation, the mimicry 
of correct responses, and the lucky guesses — all of which are good to know about when 
assessing teaching and learning. 

As the emphasis in first-year pedagogy switches from teaching to learning, from 
"what am I going to do today" to "what are they going to do today," the days of using 
only papers and exams to assess student learning are long gone. Teachers need more 
procedures capable of opening up the hidden world of learning. Think alouds can be 
helpful in this way, especially in courses emphasizing the development of cognitive skills. 
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A first-year student begins her end-of-year advising appoint- 
ment by announcing, "I can't take it any longer. This school has 
broken every single promise it made to me this year. I'm transfer- 
ring." The student itemizes a long list of grievances — a litany of 
broken promises — ranging from canceled courses to too many 
part-time instructors to residence hall problems. Their cumulative 
effect leaves the student feeling betrayed, disillusioned, and with 
the belief that departure is the only solution to her problem. 

If we are lucky, students, like the one above, do come in to 
talk to us before departing, but more often than not, they simply 
"disappear" from our campuses, leaving with the sense that they 
have not been listened to or taken seriously. This sense of "broken 
promises" is quite real for many students and a phenomenon that 
can be empirically investigated as part of a first-year assessment 
effort. 



Organizational Justice and the Psychological Contract 

Over the last 40 years, research in organizational justice has 
yielded an array of theories that address the social construction of 
fairness in organizational settings (Greenberg & Cropanzano, 2001). 
One particularly useful fairness theory is the psychological contract 
(Rousseau, 1995). A psychological contract is the set of beliefs an indi- 
vidual holds regarding reciprocal obligations with another party. In 
other words, a psychological contract is the by-product of a voluntary 
exchange relationship and represents an individual's unique percep- 
tion of promises made and agreed to. There is also a strong affective 
dimension to psychological contracts. Thus, promises are more than 
a claim of future intent: Promises are expressions of hope. Promises 
made and accepted offer good faith in exchange for trust. 

A Promise Audit 



Do students perceive certain programs or services at your 
institution as "promise breakers"? Do students perceive pub- 
lished policies as being at odds with institutional practice? Does 
campus culture communicate messages inconsistent with your 
mission to serve students? Are students tacitly made promises 
that college resources will not be able to fulfill? An assessment 
of student psychological contracts on campus — what we call a 
"promise audit" — requires a willingness to (a) listen authentically 
to students in order to hear the language of (un)fairness and (in) 
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justice embedded in their stories and (b) consider campus practices first and foremost from the 
student's perspective. We believe that assessing student psychological contracts challenges 
us to consider the implications of student perceptions of fairness for our campuses. 

Two critical — and related — assumptions underlie this framework for auditing promises. 
The first assumption is that student voices convey legitimacy of experience. Student voices 
communicate to administrators, faculty, and staff the collegiate experience as students have 
constructed and made meaning of it. Administrators' claims to "know better" or to know 
"what's really happening" on campus undermine the value of what students have to tell 
us, thus undermining the relationship with students. More important, students know when 
their voices have not been heard. 

The privileged position represented by the administrative point of view frames the 
second assumption underlying the promise audit: We must acknowledge that the experi- 
ence of reality from our "insider" perspective is not a reality shared by students. This is not 
to suggest that the faculty/ administrative voice has no legitimacy, but when it is the only or 
primary voice considered, it limits our understanding of student experience and compromises 
our ability to create collaborative learning relationships (which may be critical to developing 
and retaining communities of learners). 

Enacting the key assumptions of a promise audit requires a willingness to hold up the 
"fairness mirror" to our institutional practices, which is rarely a comfortable process. Thus, 
a key principle in the conduct of the promise audit is an acknowledgment that "it is the per- 
ception of mutuality, not mutuality in fact, that constitutes a psychological contract" (Rous- 
seau & Tijoriwala, 1998, p. 680). We recognize that students' interpretation of reality based 
on their interactions with our campuses are just as valid — as "real" — as our administrative, 
functional justifications of campus reality. 

Listening to Student Voices 

Given the subjective nature of the phenomenon we are describing, self-reported measures 
become the primary source of data regarding the nature of campus psychological contracts 
(Rousseau & Tijoriwala, 1998). Structured focus groups with new students are one way to 
begin auditing promises, paying special attention to the language of fairness that students 
use. Other content-oriented assessments could occur during academic advising sessions, 
residence hall/ floor meetings, exit interviews, and informal conversations with students. In 
addition to explicit references that students make to promises made and broken, educators 
must pay attention to other expressions, such as 

• "I did what I was told to do, and things didn't happen as I was told they 
would." 

• "I was told what to expect, and what I found was something different." 

• "Things changed, and I wasn't told what the changes meant." 

• "I was disappointed / hurt / angry because I was told ... ." 

• "Why bother with teaching evaluations, nobody takes them seriously. . .." 

• "I felt betrayed." 

• "(A campus contract maker) said that X would happen, but instead Y 
happened." 

A promise audit forces us to legitimate student voices. But promise audits also take us 
one step further: They force us to take a hard, self-reflective look at our campuses from the 
contextual perspective of the lived experience of students. 
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Promise Makers/Contract Makers 

From the onset, the promise audit reveals the complexity of how students experi- 
ence our campuses. "Contract makers" abound, often making it difficult to construct an 
integrated, coherent message for students. Do you know who — or what — are the principal 
contract makers on your campus? What promises are they making? Are these promises 
that the organization can realistically keep? Contract makers in postsecondary settings 
might include (but are certainly not limited to): 

• College catalogs and brochures 

• Schedules of classes 

• Course syllabi 

• Admissions representatives 

• Student tour guides 

• Orientation leaders 

• Current students and alumni 

• Faculty 

• Staff 

• Deans and department chairs 

• Observable campus practices and procedures 

Shared Perception 

As a simple audit exercise, a campus might start with focus groups of promise/ 
contract makers (those people on campus whom students interact with for information, 
advice, support, and teaching) and students willing to honestly participate in a promise 
audit by frankly talking about their campus experiences. Student volunteers might first 
explicate their perceptions of problematic campus issues followed by similar explication 
by the promise/ contract makers. This process would provide an opportunity to compare 
similarities and differences in perception and provide practice at "hearing" what students 
have to say. Focus groups might consider: 

• Teaching quality 

• Fairness in classroom practices 

• Course availability (scheduling) 

• Reasonable and equitable policies and practices 

• Ease of access to classes (undue policy restrictions) 

• Residence hall policies 

• Ease of access to support services 

• Quality of support services 

• Representations of campus practices vs. the reality of campus practices 



University Justice 

Psychological contracts are complex: They are a function of individual predisposition 
and cognition, social context, and organizational distinctiveness. "What's fair is fair" may be 
a truism, but what is fair on one campus may not be what is fair on another. Indeed, what 
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is fair on one part of campus may not be fair on another part of campus. Our willingness 
to assess perceptions of fairness contributes to our understanding of the student experience 
and speaks volumes about our orientation toward the ethical treatment of our students. 

An article on the "just university" in The Chronicle of Higher Education, Katz (2002) ad- 
dresses a modern tendency of the academy to all too willingly separate knowledge from 
morality, to simply describe what is rather than suggest what should be. Yet, we believe 
that those who study fairness issues in higher education have a responsibility to integrate 
description and prescription. The discovery that students perceive unfairness or injustices 
on our campus should be more than just an interesting academic exercise — it should be a 
moral framework for self-reflection and, when needed, for organizational change. 
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A Case Study on Developing Faculty Buy-In for Assessment 



Lissa Yogan Assessment of a first-year general education course at a 

small, private university was initiated in response to an external 
requirement. In 1996, an external grant provided funding to 
develop a year-long, required, interdisciplinary, 10-credit, "core" 
course. The grant-maker required serious attention to assess- 
ment. In addition to the familiar summative assessment, which 
evaluates successes and failures of a fully implemented program, 
the grant-maker asked the university to engage in formative as- 
sessment. Qualitative methods proved critical in accomplishing 
this within the prevalent institutional culture. 

This narrative briefly discusses the assessment climate 
on campus, describes how core faculty became proponents of 
assessment, explains how the core survey came to be housed 
in a larger student satisfaction survey, and outlines key ele- 
ments of a successful assessment project. 

Assessment Climate on Campus 

Since instructional faculty at this campus often dismissed 
assessment as a dubious bureaucratic exercise, instituting 
meaningful formative assessment meant a change in culture. 
Assessment itself was not new, but formative assessment de- 
signed for faculty use was. Faculty members typically cited 
their overwhelming teaching loads or a belief that "numbers" 
could not really measure the processes of teaching and learning 
as reasons not to become involved in classroom assessment. 

The reasons faculty give for not doing formal, quantitative 
assessments are typical, differing little from campus to campus. 
Schilling and Schilling (1998, p. 17 - 22) report that all concerns 
can be reduced to a list of 12: 



We already do it. 

The data will be misused. 

I'm afraid of change. 

The criteria are unclear. 

Assessment violates my academic freedom. 
Assessment is inconsistent with academic values. 
Faculty lack knowledge of assessment. 

I have no confidence of assessment. 

Too often, what is tested becomes what is valued. 
We don't need more bureaucracy. 

My plate is too full. 
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• Nobody told me about the shift from teachers and teaching to students 
and learning. 

The faculty at this institution voiced almost all of these reasons, yet the team 
did not let these typical concerns deter their efforts. 

Key Step #1: Creating an Assessment Team 

A team of several core faculty, the university's institutional researcher, and two 
faculty consultants took up the challenge of developing a formative core assessment 
program. Senior, tenured faculty and more recently hired faculty sat together on the 
Core Steering Committee and made final decisions on texts and syllabi for this impor- 
tant new University program. The research team was composed of an independent 
higher education evaluator who taught part-time in the core program, a sociology 
professor who formerly worked as a survey research consultant, and the University's 
institutional researcher. 

The survey questions related to the core were housed in a larger instrument 
that attempted to capture the essence of students' first-year experience. However, 
since the core program had provided the impetus for the assessment project, the 
research team agreed that core faculty would be the primary audience for assess- 
ment results and be the first to receive reports. Secondary audiences would be the 
university retention committee, the Provost's Office, the assessment committee, the 
teaching and learning center, the chapel council, the honor's college, and the Athlet- 
ics Department. 

Key Step #2: Using Qualitative Methods to Overcome Faculty Resistance 

The assessment team's first step was discovering student and faculty impressions 
of the former first-year required courses, as well as hopes and fears for the new core 
course and experience. Deliberately, this process was not rushed. Over the course of 
eight months, qualitative methods were employed to gather the needed information. 
The research team felt strongly that it was best to begin by listening. They did this at 
times convenient to the respondents. No faculty member had to give up designated 
work time to talk with the researchers. Students were invited to talk at times that did 
not conflict with class schedules. 

Some of the student comments took on special meanings and were repeated by 
the faculty or repeatedly used to define specific concepts. Indeed the presentation of 
the qualitative student data gave rise to faculty discussion on assessment that was 
enthusiastic and enjoyable. 

In this way, the voices of articulate students helped ease the faculty into the as- 
sessment process. As the faculty members voiced their hopes and fears about this new 
program in small groups and in a larger group discussion, they unwittingly began to 
shape the final questionnaire and join the research team. 

Core faculty posed numerous questions, and by fall 1999, agreed that a written 
survey instrument would provide more representative information than interviews. 
The culture surrounding assessment was beginning to change. The faculty had moved 
from negative attitudes toward research to a position of desiring more broad-based 
assessment tools. 
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Key Step #3: Creating the Research Instrument 

The decision was made to develop a survey in-house. The goals of the survey were two 
fold: (a) to assess the core program and (b) to assess student satisfaction with their first-year 
experience. 

Because the survey was designed to serve two primary groups — core faculty and mem- 
bers of the Retention Committee — care was taken to make sure both groups saw and were 
able to comment on consecutive drafts of the survey. This process was time consuming. The 
creation of the instrument took four months and mandated numerous revisions. 

The member of the research team charged with survey design met with a diverse group 
of people. They included members of residence life, representatives of the campus chapel, 
a student-athlete advisor, an engineering professor, the director of multicultural affairs, the 
dean of students, an employee who worked in financial aid, the head of the academic help 
program, and students in a spring semester general education class. The involvement of so 
many groups undoubtedly contributed to the survey's length but also established a link 
among faculty and staff and students. 

Each group represented by a staff or faculty member were at first somewhat territo- 
rial, concerned almost solely with "their section." Thus an early draft of the questionnaire 
missed one of the most talked-about components of first-year college life: food! Students 
in the general education class quickly pointed out this oversight, underscoring the need to 
include students in all phases of the project. 

Key Step #4: Delivering the Results and Re-involving Faculty 

Nearly 600 students (90% response rate) completed a questionnaire that asked 
150 questions about their first-year experience. Focus groups involving 33 first-year 
students were held in late spring, using questions similar to those of the previous 
year's interviews. The assessment team also provided faculty with grade distributions 
by section, gender, and race and worked on a course evaluation with questions about 
specific texts and class exercises. 

Because some faculty were not receptive to quantitative measurements of teach- 
ing and learning and others were ambivalent about "numbers-oriented" assessment, 
much care was taken to deliver the results of the survey and other assessment efforts 
in a way that faculty could "own" and use. The initial results were given during the 
end-of-the-year faculty workshop. This is a two-day workshop at which core faculty 
gather to talk about the experience of the core program and plan for the following 
year's changes. 

The presentation was interactive and given without a lot of statistical language. 
Care was taken not to provide too much information too quickly. No statistical mod- 
els were presented. Lively discussions took place over questions and results. Faculty 
ultimately focused in on two or three items that they were interested in working on for 
the following year. They did this in faculty cohort groups. 

In addition to the large group presentation, each instructor was given a printout 
of his or her results and how those results compared to the overall group means. In- 
dividual faculty members were allowed to use these results however they saw fit. The 
freedom to respond in a variety of ways allowed assessment to be seen as less heavy- 
handed and more accommodating to faculty needs, time constraints, level of knowledge 
about measurement and numbers, and individual philosophies regarding the nature 
of teaching and learning. 
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At the end of the presentation, the faculty asked questions and provided thought- 
ful suggestions for analytical methods. A core faculty member from psychology, with a 
background in structural equation modeling using survey data, joined the assessment 
team to help with analysis. He and more than 30 core faculty from various departments 
are now stakeholders in the assessment process. 

Conclusions 

Key elements of this project's success in developing faculty as stakeholders in as- 
sessment include: (a) creating an assessment team consisting of staff and faculty, across 
disciplines, and from insider and outsider perspectives; (b) using qualitative methods to 
develop trust and further collaboration by listening to and respecting student and faculty 
viewpoints; (c) following the natural pace and rhythm of faculty (meetings after graduation, 
for example); (d) addressing faculty concerns and building consensus behind the scenes; 
and (e) once they become stakeholders, letting faculty adapt an assessment program to 
meet their needs. 
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"The particular is always more than a match for the uni- 
versal; the universal always has to accommodate itself to the 
particular." Goethe 



Project Goals and Design 



The Freshman Year Experience group at Virginia Com- 
mon Wealth University (VCU) in an effort to delve into the 
specifics of first-year students' fall semester, undertook a 
large-scale qualitative project — a case study of the entering 
first-year class. VCU's first-year attrition rate was holding 
steady at 17%. Our challenge, issued by the University Re- 
tention Committee, was to find strategies for lowering that 
percentage. Our response was to interview more than 700 
first-year students about various facets of their academic and 
non-academic lives at VCU. Each week, students in required 
first-year writing courses were given 10 minutes to respond 
to an impromptu, open-ended question eliciting information 
about a particular aspect of the first-semester experience. The 
responses were spontaneous and often unpredictable. The 
project was based on a sound qualitative and ethnographic 
principle: We let our informants direct our gaze and avoided 
foregone conclusions. 

All but two of the prompts asked for anonymous respons- 
es. All prompts asked students to check off demographic in- 
formation: gender, ethnicity, place of residence, school, major, 
and composition course section. An average of 700 responses 
came in each week, generating approximately 10,000 responses 
overall. Though students' responses sometimes included que- 
ries about our sincerity (e.g., "Are you reading this?"), their 
responses were almost unanimously thoughtful and frank. 



Twenty-eight readers — faculty, staff, and administrators 
with a range of concerns and interests — analyzed the data. 
Teams of three to five readers, formed on the basis of institu- 
tional area and interest, read each prompt. Some readers with 
a range of interests read several sets of prompts. They coded 
data according to recurring patterns, which the team identi- 
fied and agreed on through a preliminary reading. Readers 
also noted important anomalies to the patterns. 



Data Analysis 
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When teams finished reading, they drew summative conclusions as to what each set 
of responses was telling them. These conclusions led to recommendations for institutional 
changes, many of which were enacted. The reading was time-consuming and the coding 
tedious, but readers unanimously agreed that the findings were worth the work. 

Working with such a large, rich data set is obviously quite messy. But most of us who 
read vividly remember comments that resonated with us, and most would agree that the 
project enhanced our ability to picture more specifically the world our first-year students 
enter. And while we were most intrigued by the detail and voices of the prompts, we assessed 
the data quantitatively, treating coded patterns numerically and breaking responses to each 
prompt down according to the demographic data collected. This quantitative breakdown 
has allowed us to examine the data from a range of perspectives and to support our many 
claims about general patterns readers drew from the data. 

For example, students unanimously praised the diversity at VCU and the opportu- 
nities to engage in diverse experiences, whether they defined this as ethnicity, academic 
discipline, or life style. Simultaneously, students made clear that ethnic relations were 
often misleadingly cordial in the classroom. Once students left the classroom, separation 
by ethnicity was the norm. 

Another discovery was that students' use of VCU's many, well-developed support 
services was quite low. Whether out of ignorance or choice, students were persisting in 
meeting the emotional, psychological, and health challenges of their first college year alone 
or with friends, sometimes successfully and sometimes not. 

Now we might have reached the same statistical conclusions through a well-con- 
structed survey, but we would have missed some of the most important and influential 
experiences in our students' lives. We were disarmed by the nature, frequency, and some- 
times seriousness of the non-academic challenges students face. We would not have been 
able to name the challenges had we simply collected bubble-test response data. We were 
able to identify not only what services students least use but also why students did not 
use them. And in some cases, responses to a bubble test might have been misleading. For 
example, in the case of ethnic relations, a bubble-test might well have indicated that relations 
were fine. We would not have learned that while relations were generally cordial, students 
were aware, often concerned, that the comfort and communication across ethnicities in the 
classroom did not reflect the community beyond the classroom. 

One very major discovery for us was the strength of feeling students had about their 
large classes. Whatever the prompt, some reference to frustration with large classes seemed 
to emerge. In a follow-up interview a year later, one student captured the main concern of 
many comments from the original study: "Teaching takes place in large classes; learning 
doesn't." Because of students' persistent references, the large class came into focus for the 
university as a site with issues that needed addressing. 

Why Carry Out This Kind of Research? 

Colleges and universities need to hear a full range of student voices. This type of re- 
search allows every respondent to have his or her own voice and creates a broader base of 
both input and output (as readers spread the word). Because of the power of student voices, 
this is research that engages. It is also ultimately replicable research: Virtually everyone 
can tap first-year composition programs, and each campus can create its own demographic 
categories for identifying how its important subpopulations differ from the whole. 

Understanding generational shifts between those who design programs and services 
and those who use them is important, perhaps crucial, if programs and services are to 
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meet best the needs of those they serve. Professors and administrators are perpetu- 
ally graying and giving way to new generations coming on board. Students' habits 
of learning, their concerns, and cultural visions are constantly changing. For all of 
us. The Prompts Project shook up some of our assumptions and gave us new eyes. 
We were able to begin defining agendas for change that were important and relevant 
to our students. Our students are never, in any way, a generic population. 

The First- Year Prompts Project Questions, Fall Semester 1990 

Week 1 New Student Orientation was last week. If you attended, what about the 
orientation was beneficial for you and what was not? If you did not attend, 
why not? If you are no longer in your first year, but attended orientation 
in the past, we'd like to hear what you remember of it as well. 

Week 2 You're here at VCU. You've worked hard to get here. Write for 10 minutes 
about your hopes, dreams, fears, and expectations for this semester. 
Week 3 Of everything you've seen, heard, done, or had happen to you during 
these first two weeks at VCU, what has been most striking? 

Week 4 When we speak of personal space [a concept introduced in a course read- 
ing by Brent Staples], we speak of people or things intruding on it, of 
having enough or not, of being comfortable in our space or not. People, 
places, sights, sounds, and actions — all of these affect our 'space' just as 
we each affect the space of others. What is your space like at VCU? 

Week 5 In English 101, one of the readings is 'Dangerous Parties.' What is 'party- 
ing' like here at VCU? What role do you think 'partying' plays here? How 
do you feel about it? 

Week 6 Classes have now been in session for a little more than a month. How are 
you feeling about being a student at VCU? If you're feeling positive, what 
has contributed to your comfort here? If you're feeling less than positive, 
what's missing for you? 

Week 7 Many of you commented on the diversity of the student community at 
VCU, which has long been a distinctive characteristic of this university. 
As a VCU student, how would you react to this statement: 'Students of 
various racial and ethnic backgrounds get along well'? 

Week 8 Most English 101 classes have been reading about AIDS. In addition to 
HIV infections, we know statistics about some other sexually transmitted 
diseases (STDs). For instance, about 10% of college woman have chla- 
mydia, which often has no obvious symptoms. Last year alone, almost 
1,700 visits to the Student Health Services were related to STDs. Recent 
estimates say that by 18 years of age, 78% of women and 86% of men are 
sexually active. We know that college students feel little to no sense of 
personal risk, though these statistics are alarming. Since much is said 
about 'safe sex' methods, there seems to be a discrepancy between risk 
and practice. What do you think — why is the use of condoms relatively 
low? 

Week 9 Many of the prompts thus far have been about your overall experiences 
at VCU or about specific issues like 'personal space' or 'partying.' Now 
we'd like you to turn to a more academic topic. It's midterm. Do you 
know what your grades are? What kind of feedback have you gotten so 
far about how successful you've been in your courses? 
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Week 10 This week's section in the English 101 reader is entitled 'Educational Short 
Cuts: Are We Cheating Ourselves?' Here are some actual scenarios from VCU 
Academic Integrity Policy cases: 

1. Two students submit term papers for the same class and, upon reading 
them, the instructor notices that many of the quotes and other copy are 
exactly the same. 

2. A student submits an exam paper which has the same answers as the 
instructor's exam key, including the mistakes that the instructor has made 
in solving the problems. 

3. An instructor has placed materials needed for a class assignment on reserve 
in the library. A student in the class, who delayed working on the assign- 
ment, improperly removes the material from the library so that it is no longer 
available to the other students in the class. 

What kind of academic dishonesty have you either observed or participated in 
here at VCU? What do you think the university should do about this problem? 
Week 11 With nearly 22,000 students, VCU is a big school. The general folklore about 
college differences is that students at small schools have better communication 
with their teachers than do students at large schools. What is communication 
like between you and your instructors? Also, if you could have a special rela- 
tionship with one faculty member (your advisor or someone else), what would 
you like it to be? 1 

Week 12 As you know, life does not always go smoothly. Things like these actually hap- 
pen to many students: 

1. A friend reacts to the relative freedom of college by becoming heavily in- 
volved with alcohol. 

2. A phone call comes from home: parents have decided to separate, intending 
to divorce as quickly as possible. 

3. We see on the news where a VCU-connected person is missing or held hos- 
tage and feel angry and helpless. 

What kinds of serious crises have you and your friends here experienced this 
semester, and how have you/ they coped?" 

Week 13 The semester is moving toward a close, and you are probably making academic 
plans for 1991. Has anything happened academically or otherwise that has made 
you question your ability to succeed/ remain at VCU? 

Week 14 During week 14 we returned to students their responses to Prompt 2 and asked 

them to "Take a look at what you wrote at the beginning of this semester. Com- 
pare your thoughts then with the realities you've experienced." 2 
Week 15 Ideally, colleges and universities encourage a lively community in which all 
members — students, faculty, and staff — can discuss ideas, both in and outside 
the classroom, with a common goal of learning. Have you found VCU a place 
where you can take part in that kind of dialogue? If so, in what ways has that 
happened? If not, what suggestions would you make for the university? 

Impact 

It is all too common for reports of assessment findings to gather dust on the shelf. The 
report resulting from The Prompts Project, "The Challenge of the First Year," led instead to a 
series of discussions and actions that continue to this day. 

The first effect, though, was evident even before Hodges wrote the report: The readers. 
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who had been "hearing" this environmental scan in students' own words, quickly began 
talking about what the students were telling us. These readers took students' voices into 
meetings about then-present issues and into planning sessions about the future, thus let- 
ting students themselves have a voice as advocates for change. 

Leaders of The Prompts Project took student comments about large classes and devel- 
oped a grant proposal, "Improving Student Success and Satisfaction in the Large Classroom 
Environment." Funded by the State Council for Higher Education for Virginia's Funds for 
Excellence Program, the grant undertook four pilot projects that tried in various ways to 
ameliorate the negative aspects of large classes. Several of the pilot efforts — Supplemental 
Instruction, use of technology to increase faculty-student communications and feedback, 
and Freshman Interest Groups — were successful enough to receive permanent funding 
from the university. Another effort originating in The Prompts Project was a summer in- 
stitute for faculty teaching large course sections. During the institute, we discussed how 
Chickering and Gamson's (1987) "Seven Principles for Good Practice in Undergraduate 
Education" could enhance large-classroom settings. Faculty implemented these ideas the 
following academic year. 

Individual units around campus used the findings to change intervention strategies. 
For example, the student health service put far more emphasis on social marketing after 
seeing students' responses. People continue to refer to The Prompts Project in current 
retention discussions, even though a decade has passed since the study. And because we 
learned so much of value from this project, we will soon implement a similar study. The 
Voices Project, with closer focus on our first- and second-year students' academic issues. 

Notes 

1 This final sentence is admittedly unfortunate in its phrasing. 

2 The idea for this prompt originated with research by Miami University, Ohio. 
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Assessment is becoming increasingly important on most cam- 
pus, but as Carmean (2003) convincingly demonstrates in outlining 
"ten barriers to successful assessment initiatives," it also carries a 
great deal of negative baggage. Angelo (1999) noted nearly five 
years ago that "Today, most faculty and academic administrators 
have finally, if reluctantly, come to accept that dealing with both 
[accountability and improvement] is a political and an economic 
inevitability" (p. 1). Huba and Freed (2000), however, remind us 
that faculty should be committed to "assessing for improvement 
rather than accountability" (p. 87). Assessment for both reasons is 
certainly a common practice at the US Air Force Academy (USAFA). 
For example, all departments routinely prepare unit self-assess- 
ments focused on their core courses and their impact on USAFA' s 
seven educational outcomes, reinforcing a close alignment between 
individual courses and broader mission elements. These educa- 
tional outcomes include critical thinking, intellectual curiosity, and 
cooperative and independent learning. Because of this depart- 
mental and institutional commitment to viable assessment, there 
is a strong need for expertise and assistance. In many cases, the 
Center for Educational Excellence (CEE), USAFA' s unit responsible 
for faculty development, assessment, academic technology, and 
research, has been increasingly involved in helping departments 
gather and interpret both quantitative and qualitative data. 

The USAFA has developed a unique process, interactive 
focus groups, to acquire qualitative data from those most directly 
affected by curriculum and pedagogical transformations — the stu- 
dents. Focus groups have been used in academia for some time, 
and Libby Morris's essay in this volume outlines some of the more 
common approaches to focus groups. However, the USAFA focus 
group model incorporates a number of unique elements, making 
this a novel approach to qualitative assessment. 



An Overview of the USAFA Focus Group Model 

The focus group model developed at USAFA employs a variety 
of techniques to maximize data collection from as many students as 
possible. An optional survey, an index card activity, and a group- 
based roundtable ranking activity, in addition to the series of open- 
ended questions make this model unique. This highly structured, 
yet efficient, protocol has been widely embraced by faculty members, 
course directors, and departments despite their varying assessment 
needs. For simplicity sake, we refer to all these parties as "clients." 



55 



56 Millis 



The focus group protocol is typically used with student volunteers who come to a neu- 
tral location to share ideas about their courses. But, in many instances, because class sizes 
are typically under 20 students, we will use the same protocol with all students enrolled in a 
given course. To assess, for instance, the effectiveness of a new core law course, all students 
enrolled in the three pilot sections participated in a focus group conducted by a CEE repre- 
sentative and a member of the law department who knew the course content, but who was 
not teaching the course at the time. 

The focus groups are organized systematically to collect as much data as possible within a 
50- to 60-minute time limit and so that long-term comparisons are possible as courses or programs 
are modified. Rossi and Freeman (1993) emphasize that decisions should be based on the con- 
vergent patterns in the evidence, much as a courtroom jury must sift through complex and often 
conflicting evidence to reach a verdict. Similarly, Nyquist and Wulff (2001) advocate a research 
perspective when working with faculty on course or program improvement because "the ap- 
proach is both systematic and complete and also familiar and appealing to the client" (p. 46). 

Meeting with the Focus Group "Client" 

All focus groups begin with a client-centered discussion to explore the course objectives 
and the objectives for the focus group. Included in this discussion are questions and activi- 
ties that would provide valuable data and the logistics involved in selecting participants, 
setting up the room, and providing refreshments, if any. We typically share with clients the 
questions used in previous focus group sessions. Often, clients will have similar concerns 
such as the grading system or the textbooks. In other cases, the questions will be unique 
to a specific course or approach. For example, first-year engineering students expecting 
passive lectures suddenly learn that they will be actively involved in problem solving and 
group interactions. They consider these demands stressful. Thus, instructors for a first-year, 
problem-based engineering course were interested in what Woods (1994) called the "grieving 
process" for coping with change in the academic environment (pp. 1.1-1. 2). 

Until USAFA opened a video teleconferencing facility (VTC) — where audiotaping is easy 
and almost fool-proof — the CEE used a paired approach with one staff member responsible 
for conducting the actual session (the facilitator) and another staff member responsible for 
"logistics," including setting up and monitoring portable audiotaping equipment and making 
notes of responses to open-ended questions. Now that focus groups routinely take place in 
the VTC, only one facilitator is needed because the equipment is "push-button" ready and 
hand-generated back-up transcriptions are not needed. 

The transcription preparation process has been simplified through the use of a medical 
transcription service. This service can turn around a one-hour tape in a day at a cost of about 
$75, money well spent given the need for rapid feedback. Because the service returns the 
transcript as an e-mail attachment, we also have the ability to correct errors and omissions. 

Components of the Focus Group Protocol 

The Optional Student Survey 

Depending on the client's objectives, the students may receive a survey as they arrive for the 
focus group, a practice that reinforces the seriousness of the project and helps students feel at ease 
when they have an initial task to perform. The questions on the survey are often idiosyncratic, 
such as asking students how much time they devote to the course. The surveys are collected 
and the responses tabulated anonymously. 
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Instructions for Students 

After the expected students arrive, the facilitator explains the nature of the focus group 
session, including the ground rules for discussion, while emphasizing the confidentiality of 
responses and describing how they will be used in reporting. Students are also made aware 
that the session will be audiotaped. To ensure confidentiality, each student is assigned a 
number for identification purposes. Students quickly begin saying things such as: "This is 
Student Ten. I agree with Number Three's idea, but I think that Number Twelve's proposed 
changes to the final exam will cause greater learning and less grief for students." 

The Initial Focus Group Activity: Ratings on Individual Index Cards 

After the students complete the survey, if one is used, and hear the introductory remarks, 
they are handed an index card. Working independently, students jot down on the index card 
(which also includes their assigned response number) a word or phrase to describe the course 
(or program) and a number from 1 to 5, with 5 indicating a high level of satisfaction with the 
course (or program). Usually the facilitator has the students indicate, roundrobin fashion, 
their responses. Typical comments might be: "This is Student Six: I gave the course a five and 
an 'Awesome'" to "This is Student Eight. I gave this geography course a number one. My 
phrase is: 'As dry as eating saltines in the Sahara.'" This pubic disclosure serves a number of 
functions. First, it enables students to feel comfortable with their peers. Because everyone has 
candidly revealed his or her impressions, no one needs to "second guess" another student's 
perceptions. The facilitator is also quickly able to determines the overall "climate" of the 
group — knowledge that can help him/her formulate probing questions. Finally, the open dis- 
closure establishes an atmosphere of trust before the facilitator asks open-ended questions. 

This focus group model proves useful for both course and program assessment. For exam- 
ple, to learn more about the management major, we gathered data on the program by conducting 
focus groups in the management capstone courses over a two-year period. In this case, the stu- 
dents rated the entire major and offered descriptive words to support their assessments. During 
this two-year period, the department was "closing the loop" by making changes suggested by 
the focus groups and other data sources and then measuring the impact of those changes. This 
assessment process also assisted in the department's successful AACSB accreditation. 

Similarly, after the initiation of a first-year experience course piloted in Fall 2002, the CEE 
conducted focus groups with the first-year students enrolled in this course. This assessment 
practice was embedded in the project from its inception, a practice recommended by other 
faculty development/ assessment experts familiar with other first-year experience courses 
(Stassen, 2000, p. 274). The focus groups revealed some "pluses," such as the perceived rel- 
evance of the course, which focused on a real-world scenario (the Gulf War) from multiple 
disciplinary perspectives. It also uncovered some weaknesses such as a workload students 
perceived as too demanding for the credit given. 

The Open-Ended Questions 

Following the index card activity, open-ended questions are posed by the facilitator, who 
has discussed with the client beforehand which questions all students should answer and which 
ones are suitable for random responses. Typically, the open-ended questions target issues of 
particular concern to the client. For example, the Taw Department was interested in the value 
of a new textbook, while an English professor wanted to understand the impact of one-on-one 
grading conferences that had replaced traditionally marked papers. A course director for a basic 
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physics course wanted to learn how students responded to "Preflight Exercises," electronic 
homework assignments professors used to tailor lectures and class activities based on dem- 
onstrated student knowledge. Generally, the facilitator asks 8 to 10 questions during this 
segment of the focus group. 

A Second Activity: Ronndtable/Ranking 

The Roundtable / Ranking, a group activity, also makes this focus group protocol differ- 
ent from traditional models. The facilitator places the students in small groups for this exer- 
cise. Each group receives a handout with specific instructions to brainstorm all the strengths 
of the course and then to brainstorm all the things about the course that could be improved 
(See the Appendix for sample brainstorming sheets.). The paper circulates rapidly from one 
student to another as each writes down an idea and says it aloud. These brainstorms are 
done quickly so students do not get into an analytical mode. During the next phase of this 
activity, each group rank-orders the strengths of the course and then the weaknesses. This 
rank-ordering step is critical because it enables students to reach consensus on their priori- 
ties and to eliminate any idiosyncratic responses. The roundtable and ranking exercise take 
only 10 minutes out of the 50 to 60 minutes available. If time permits, each group can be 
called upon to report its top three strengths and top three weaknesses, but this step is often 
omitted to allow more time for the concluding open-ended questions. 

The Client Reports 

All clients receive summary report which includes (a) survey results, if a survey was ad- 
ministered; (b) a histogram detailing the index card responses; and (c) a color-coded table of 
the Roundtable /Ranking data. The histogram (Figure 1) offers a visual representation of the 
rankings of the course or program and provides a list of the descriptive words or phrases (e.g., 
"Stimulating," "Limp," "A Definite Challenge") associated with those rankings. In addition to 
highlighting current levels of student satisfaction, the histogram provides useful longitudinal 
data. For example, successful course improvements would result in a rise in the "four" and 
"five" columns over time. 
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Figure 1 . Histogram of Index Activity for Management 210 Focus Group (November, 2001), where the height 
of the columns indicates the frequency of response for a particular satisfaction rating. The words inside the 
columns are the descriptors associated with that ranking. 
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The Roundtable /Ranking data are displayed in a color-coded table (Figure 2), which 
indicates the top three strengths and weaknesses for a course or program identified by a par- 
ticular focus group. A CEE assessment expert identifies trends through a quick cluster analysis 
and color-codes recurrent themes systematically. For example, items related to the work- 
load — whether identified as strength or weaknesses by any team — might appear in orange. 
Textbook-related comments might be coded in blue. We deliberately use the same colors for 
similar items in every report: This color consistency makes changes over time easy to spot. 
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Figure 2. Roundtable / Ranking Activity for Management 210 Focus Group (November, 2001) show- 
ing thematic representation of course strengths and weaknessess. 

Quick Course Diagnosis (QCD) 

Over time, the demand for focus groups has skyrocketed, but many clients also wanted 
to have the focus group data without devoting an entire class period to data collection. To 
address this need and staffing constraints, an abbreviated version of the focus group protocol, 
known as a Quick Course Diagnosis (QCD) was developed. The QCD method involves a 
single facilitator who conducts only the Index Card and the Roundtable /Ranking activities. 
The entire process can be completed in 15 to 20 minutes. To augment this data, the class 
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participates in a five-minute group-consensus activity, determining the top three strengths and 
"weaknesses" (things to improve) of the course. A student or the CEE facilitator records the 
consensus-building process, giving CEE a permanent record of this portion of the QCD. This 
information is also included in the final client report, similar to the one described above. 

QCDs are not as data-rich as the focus groups because of the omission of open-ended 
questions; however, the three reports prove extremely valuable. This trade-off results in less 
time devoted to classroom administration and decreased costs because there is no need for the 
transcription service. 



The Client/CEE Feedback Session 

Rather than simply hand-off the focus group or QCD reports, CEE staff members prefer 
to meet with their clients to discuss the assessment results. During this meeting, we encourage 
reflection and appropriate interpretation of the data and explore possible ways to strengthen a 
course, program, or major. The data provide a solid foundation for informed, research-based 
decision-making. 



Conclusion 

Students are asked to comment on the value of the focus group process at the close of each 
session. Without exception, all cadet focus groups have indicated that they like the purpose and 
format of these structured interviews. They enjoy the informal interactions far more than paper- 
and-pencil surveys or bubble sheets, value that their opinions matter, and express concerns about 
the impact of their responses on the Academy. 

The depth of focus group data far exceeds that of surveys or end-of-course critiques and has 
been invaluable in course and program improvement. For example, the law department made 
significant changes in their course structure and content, the textbook, and evaluation methods. 
A core course linking geography and meteorology adopted new texts and activities that fostered 
integration. Based on focus group responses, faculty are also incorporating active and coopera- 
tive learning approaches into the course. An experimental first-year course in problem-based 
engineering added more "scaffolding" for students over a two-year period. 

Focus groups remind us of Parker Palmer's (1998) words: "Good talk about good teaching 
can take many forms and involve many conversation partners — and it can transform teaching 
and learning. But it will happen only if leaders expect it, invite it, and provide hospitable space 
for the conversation to occur" (p. 160). Structured focus groups allow those conversations to 
take place. 
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Index card numbers (top right corner): 

Roundtable Activity # 1 

Passing this sheet of paper rapidly from one person to another, please jot down all of the relevant 
strengths of the course, saying them aloud as you write. 



Working as a team, rank order the strengths you identified, with the most important ones at the top 
of your list. Rank at least three by writing the numbers "1," "2," and "3" next to the strengths you 
identified. 



Roundtable Activity # 2 

Passing this sheet of paper rapidly from one person to another, please jot down all of the 
"negatives" of the course — the things you would change, saying them aloud as you write. 



Working as a team, rank order the weaknesses you identified, with the most significant ones at the 
top of your list. Rank at least three by writing the numbers "1," "2," and "3" next to the weaknesses 
you identified. 



The College Classroom Environment Scale 



For the majority of students at Portland State University, the 
general education program. University Studies, is based upon 
core learning goals and objectives. The goals of the program are 
accomplished through a four-year, interdisciplinary curriculum 
based on learning communities. Creating community on college 
and university campuses has been particularly explicit goal for 
higher education since the publication in 1990 of Campus Life: In 
Search of Community (The Carnegie Foundation for the Advance- 
ment of Teaching). 

The first-year course. Freshman Inquiry, is the foundation 
of the program. It is a year-long course developed by an inter- 
disciplinary team of faculty paired with undergraduate student 
mentors. We have found that without community building many 
of the program goals are difficult to achieve. While the concept 
of community has obvious co-curricular dimensions and implica- 
tions, it is reasonable to assume that one of its primary locations 
on campus is the classroom. Evaluating this construct within the 
classroom is a challenge, calling for both qualitative and quantita- 
tive approaches. 

In 1994 we began using the College Classroom Environment 
Scale (CCES), developed by Roger Winston at the University of 
Georgia, as one on our primary measures of students' perception 
of the community environment in the classroom. Eight years of use 
and the development of more external and internal comparative 
data have enhanced the instrument's utility and credibility for us. It 
is now an expected and accepted part of our assessment process. 

The purpose of the CCES is to develop information about 
student perceptions of the academic environment at both the class- 
room level and larger organization levels. The CCES consists of 62 
items on a Likert scale, forming six subscales or constructs. These 
are as follows: 

1. Cathectic Learning Climate (CLC). This scale indicates 
a charged academic atmosphere that stimulates 
students to be active class participants and to seek 
classmates' opinions and reactions. 

2. Professorial Concern (PC). This scale indicates to what 
extent students perceive the instructor as being per- 
sonally concerned about them as individuals. Higher 
scores on this scale suggest students see the profes- 
sor as being friendly, caring, open, empathetic, and 
respectful. 
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3. Affiliation (AF). This scale indicates students' perceptions of numerous informal 

interactions with each other. It also checks to see if a supportive and friendly 
peer atmosphere exists. Cooperation and development of mature interpersonal 
relationships are perceived by students as valuable. 

4. Academic Rigor (AR). This scale is descriptive of an environment that is perceived 

as intellectually challenging and demanding. Students perceive excellence 
and personal responsibility as the norm, which are expressed through high 
but realistic evaluation standards. 

5. Structure (ST). This scale describes an environment where students perceive 
evaluation criteria and syllabi as clearly articulated and followed. There is 
little ambiguity concerning assignments or expectations. The instructor is 
viewed as an authority in the course content. 

6. Inimical Ambiance (IA). This scale is negatively scored and is the only subscale 
where a low score is desirable. This scale itself describes an environment 
that students see as being hostile, highly competitive, rigidly structured, and 
one in which they are uncomfortable asking questions or giving opinions. 
Authority is perceived as arbitrary and is exercised in a dehumanizing and 
aggressive manner. 

The CCES is administered at the mid-point of winter term in every Freshman Inquiry 
course. These forms are completed in class without the faculty member present, as is done 
with the end of course evaluations. However, we found that the student mentor or some 
other person must supervise the administration to reduce external validity problems. The 
class mentor stays with the students to make sure the protocol for the survey completion is 
followed. 

Faculty receive individual reports with their scores and an explanation of the subscales. 
Many faculty were initially resistant to the assessment of the class environment and had 
problems interpreting their scores. They also wanted normative data so that they would 
have some basis for comparison; however, there was very little available in the early years. 
Winston, author of the CCES, encouraged us to develop such data for our own purposes. 
Until we were able to establish our own norms, it was difficult to convince faculty of the 
CCES's usefulness. 

In the current system, individual reports are sent to each faculty member. Also included 
is a table of aggregate scores from each year of the administration. Scores are followed by a 
brief explanation of the subscales with examples of questions used to compile the subscales. 
Faculty often look forward to finding out how their own scores change from year to year. 

Use of the CCES Data 

The CCES data is reviewed by the director and the administrative body of the program. 
Data are incorporated into the Annual Program Assessment Report. We have found that the 
CCES scores validate and complement the findings of our qualitative data collected each term. 
We use the CCES data for program improvement in the following ways: 

• Faculty who consistently score well are invited to present sessions for the 
fall and spring program retreats. They are also asked to facilitate faculty 
development sessions in order to share their successes and best practices 
with other faculty. 
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• When faculty scores are significantly lower than the aggregate, resource 
personnel are made available to work with faculty members on an individual 
basis to determine student concerns and ways to improve the climate within the 
classroom. 

• Faculty use their scores in professional portfolios for purposes of annual review, 
promotion, and tenure. 

Within the classroom, faculty use the CCES data in the following ways: 

• Because of the interdisciplinarity of the program, faculty can evaluate 
different teaching strategies they may use for presenting material in diverse 
disciplines. 

• Often faculty have used the results to discuss and clarify issues in the 
classroom directly with their students. This is helpful in two ways. Students 
feel that the assessment process has been meaningful and that they have 
a voice in the classroom. Faculty and students can work together as a 
community to resolve problems before they turn divisive in the classroom. 

• When faculty review the subscales and their results it may help them 
understand how their students perceive the class environment as opposed to 
how they themselves perceive it. This may be an eye opening experience for 
many faculty. 

• Faculty and students may use the results to celebrate the success of the 
community which has been built in their class. 

Analysis and Conclusion 

The data collected from years of using the CCES help validate the success of our Fresh- 
man Inquiry program. The following are those things we have learned from the data: 

• Our faculty aggregate scores for Professorial Concern remain the highest of all 
subscales except for one year when this subscale was tightly clustered with 
those of Structure and Academic Rigor. 

• Scores for Academic Rigor and Professorial Concern always tend to be higher 
than all others. Subscale scores for Inimical Ambiance have remained at less 
that a .20 difference for the eight years of use. This shows us that students 
continue to perceive a feeling of community within our classrooms over an 
extended amount of time. This has a direct impact on our work in sustaining 
such a unique program. 

• Faculty who teach two sections of the same inquiry course, using identical 
pedagogies and content, have found that scores are often remarkably different 
from one class to the other. This helps to explain how personalities and learning 
styles of students in a class have a direct impact on how they perceive classroom 
climate. This has resulted in consultation directly related to a particular cohort 
of students. 

• Students say that they appreciate when their instructor discusses the results of 
the CCES assessment with them. Students may have few comments when this 
happens, but it is the acknowledgment of the importance of their participation 
that appears to hold the real value for them. Working with faculty to assist them 
in developing alternative teaching strategies has also been very successful. 
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• The CCES data help inform us about areas of need within the program. We 
can then concentrate on topics that address the issues raised for our faculty 
development program, biannual retreats, and particularly our new faculty 
orientation program. 

• The data identify faculty who excel at creating community in the classroom. 

Those faculty can function as mentors to new faculty and others, as well as help 
in the design of faculty development sessions to improve the overall program. 

The CCES has proved to be a valuable element of our total assessment program. 
However, it is only one part of a yearlong assessment plan. The data is used in conjunc- 
tion with course evaluations, student portfolio review results, and feedback from mentors, 
team members, and program administrators. 
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The recent surge of interest in assessing the first year of 
college has made campuses more reliant on information that 
documents students' experiences and demonstrates institutional 
effectiveness. Many first-year assessments collect data at a single 
point in time, yielding data on students' behaviors, attitudes, and 
perceptions, but alone they tell us very little about how students 
have changed since entering college and what impact the first- 
year experience has had on their development. 

To assess the impact of campus environments and col- 
lege experiences on important first-year student outcomes, 
it is critical to collect data on students at a minimum of two 
time points — when the students arrive at the college and after 
some exposure to the first-year experience. These longitudinal 
data may be used within Astin's widely known Input-Environ- 
ment-Outcome (I-E-O) model of assessment, which provides a 
framework for understanding which college environments and 
experiences contribute to which college outcomes (Astin, 1993). 
Essentially, information collected on students at the point of 
college entry may be viewed as "inputs" that shape students' 
college experiences; the impact of college "environments" on 
student "outcomes" cannot be assessed until student differences 
on these "input" characteristics are accounted for. 

The concept behind the I-E-O model has been central to the 
work of the Cooperative Institutional Research Program (CIRP) 
at UCLA. For more than 35 years, the CIRP has provided colleges 
and universities with a tool to assess students at the point of col- 
lege entry (the CIRP Freshman Survey) and has offered surveys 
that collect longitudinal follow-up data. More recently, the CIRP 
has introduced a new survey. Your First College Year (YFCY), 
specifically designed to assess students' experiences over the 
first year of college. Linking data from CIRP and YFCY enables 
institutions to conduct longitudinal assessment of the first-year 
experience. The history, purpose, and function of each of these 
two surveys are described below. In fact, CIRP and YFCY have 
emerged as the only national, longitudinal study of students over 
the first year of college. 



CIRP Freshman Survey 

The CIRP Freshman Survey was initiated in 1966 at the 
American Council on Education and has been housed since 1973 
at UCLA's Higher Education Research Institute (HERI). Each 
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year the CIRP surveys some 350,000 to 400,000 full-time students who constitute the entering 
first-year classes at a national sample of 650 to 700 institutions. All regionally accredited 
two- and four-year colleges and universities are eligible to participate in the CIRP. 

Participating colleges administer the Freshman Survey (also known as the Student 
Information Form, or SIF) at the start of first year of college, usually during orientation. 
The survey is a four-page questionnaire, appropriate for a variety of students attending all 
types of collegiate institutions. Although the national normative data published each year 
are based only on first-time, full-time first-year students, participating institutions receive 
separate reports for their part-time and transfer students. The survey is revised annually to 
reflect the changing concerns of the academic community and to address emerging research 
questions. However, the survey also repeats items from year to year to help institutions 
assess trends. Data collected from each student include: 

• Demographic characteristics, including age, gender, racial / ethnic back- 
ground, and religious affiliation, as well as parental education, income, 
and occupation 

• Secondary school background, including high school grades, high school 
courses, year graduated from high school, and level of academic involve- 
ment in high school 

• College finances, including sources of financial support and student concern 
about finances 

• Orientation toward college, including why students chose to attend college, 
why students selected a particular college, number of other college ap- 
plications, and student expectations of their college experience 

• Aspirations, including probable college major, career preferences, highest 
degree sought, and life goals 

• Attitudes and values, including personal values (e.g., helping others, achiev- 
ing recognition, being financially well-off), political orientation, and at- 
titudes toward current national issues (e.g., affirmative action, abortion, 
drugs, rights of criminals, handgun control, taxes). 

In addition, participating institutions may add up to 21 supplemental questions for 
their own students. Complete tabulations of student responses to these local questions are 
included in the summary report returned to the institution. 

Institutions use their Freshman Survey data for a variety of purposes, including: 

• Admissions and recruitment. Creating a profile of entering first-year class, iden- 
tifying new markets for recruiting activities, identifying factors that affect 
student decisions about college choice, assessing the impact of financial aid 
on college selection decisions 

• Curriculum and program planning. Assessing outreach efforts, assessing the 
academic preparation of entering students, assessing students' intended 
majors and expected campus involvement, curriculum planning and review, 
evaluating institutional policies and procedures 

• Public information. Feature articles in the campus press; press releases for local, 
regional, and national news media; articles for alumni bulletins and magazines; 
public addresses to community, government, and professional groups 

• Institutional research. Trends in the characteristics of new students; comparisons 
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between students in different schools, divisions, or departments; comparisons 
with students at institutions of similar type; comparisons with students in 
national norms; baseline data for longitudinal studies of student development 
and educational outcomes; baseline data for retention studies 
• Trends analysis. The Freshman Survey repeats items from previous years, en- 
abling institutions to assess trends in the characteristics, experiences, attitudes, 
values, and expectations of their entering first-year students. 

The oldest survey of its kind, the annual Freshman Survey also provides information 
about our national population of first-year students that is unmatched in terms of breadth 
and depth. Nationwide results are published each year in The American Freshman: National 
Norms and receive widespread attention in the national press and among researchers and 
policymakers representing a wide range of fields and interests. National data are useful to 
the general community of current and future college students, their parents, and college 
faculty as well. 



Your First College Year (YFCY) 

In 1999, the Higher Education Research Institute joined forces with the Policy Center 
on the First Year of College to develop and administer a new survey instrument titled "Your 
First College Year" (YFCY). Designed as a follow-up to the CIRP Freshman Survey, YFCY 
was conceptualized to fill a void in the nation's survey market, namely the absence of an 
instrument designed specifically to assess the first college year. Between 2000 and 2002 YFCY 
was developed and pilot tested with support provided by grants from The Pew Charitable 
Trusts and the Atlantic Philanthropies. Since 2000, more than 67,000 students at 325 institu- 
tions have participated in YFCY. 

The YFCY instrument was developed in consultation with students, faculty, and aca- 
demic and student affairs administrators at two- and four-year campuses around the country. 
Approximately one-third of the survey items on YFCY directly posttest items on the CIRP 
Freshman Survey. YFCY addresses several aspects of the first college year, including: 

• Academic achievement 

• Academic skills and engagement 

• Learning strategies and pedagogical practices 

• Residential and employment experiences 

• Interactions with peers, faculty, and staff 

• Satisfaction with curricular and extracurricular experiences 

• Patterns of behavior 

• Student values and goals 

• Self-confidence and feelings of personal success 

The instrument is available online or as a paper form. Both versions include space for 
campuses to ask up to 21 questions of local relevance. Institutions may administer the survey 
in the classroom, via campus mail, or via e-mail. Although YFCY is intended to serve as a 
follow-up to the CIRP Freshman Survey, it can be used as a stand-alone instrument. 
Institutions use their YFCY data for a variety of purposes, including: 

• Comparative analyses. Because students' responses to the survey are compared to 
national and institutional peer group aggregates, participating institutions 
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can determine where their first-year cohort "stands" relative to the experiences 
of first-year students at large. In addition, participating institutions are able 
to disaggregate responses to conduct comparisons between different groups 
of students at their campus. For example, it is possible to compare first-year 
outcomes such as adjustment or retention based on participation in a learn- 
ing community, academic "cluster" program, or a first-year seminar. It is also 
possible to analyze the data by gender, race / ethnicity, or place of residence. 
Space for institution-specific supplementary questions offers additional op- 
portunities to conduct within-institution analyses. 

• Descriptive analyses. YFCY collects information on a wide range of cognitive and 
affective measures providing comprehensive data for single- or multiple-institu- 
tion analyses of persistence, adjustment, and other first-year outcomes. These 
analyses can answer questions about the first year of college including: 

• What are students' academic experiences in the first year of college? 

• What are students' social experiences in the first year of college? 

• Which factors influence students' decisions to re-enroll for a second 
year? 

• How well do students adjust to their first year of college? 

• How do first-year students spend their time? 

• What are the values, attitudes, and goals of first-year students? 

• Longitudinal analyses. Because YFCY is designed as a follow-up survey to the 
CIRP Freshman Survey, it allows for longitudinal research on the first year 
of college. Therefore, institutions are able to use these data to evaluate the 
academic and personal development of students over the first year of college 
and to assess the impact of institutional programs, policies, and practices 
on the students' experiences and outcomes. Further, YFCY may be used in 
conjunction with local baseline data, registrar's data, or other campus-based 
assessment efforts to enhance an institution's understanding of the first-year 
experience on its campus. 

• Trends analyses. Like the CIRP Freshman Survey, YFCY repeats items from 
previous years. As such, institutions participating regularly in YFCY are able 
to assess trends in the characteristics, attitudes, values, classroom practices, 
personal behaviors, satisfaction, and adjustment of their entering first-year 
students. 



Impact on the Field of Educational Research 

The number of institutions participating in YFCY has grown each year it has been offered. 
Larger and more diverse samples of students and institutions suggest greater opportunities 
to inform a national perspective on the experiences of first-year college students. Further, 
since the majority of YFCY respondents also participate in the CIRP Freshman Survey, there 
is unprecedented opportunity to contribute to scholarship on student development over 
the first year of college. This is critical since the vast majority of research on college student 
development is based on studies of students over four (or more) years of college. The four- 
year time frame prohibits an understanding of the role that the first — and arguably the most 
pivotal — year of college plays in shaping students' capabilities, values, attitudes, percep- 
tions, and aspirations. It is our hope and expectation that the data generated by CIRP and 
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YFCY — at both the institutional and national levels — will make a major contribution to the 
policy and research dialogue on the first-year experience. 
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Survey Dajt^sts^irt of First-Year Assessment Efforts: 

Using the Cooperative Institutional Research Program (CIRP) 
Annual Freshman Survey 

T Daniel House University planners need assessment information that 

enables them to understand the characteristics of entering col- 
lege students and the relationship of those characteristics to 
subsequent college outcomes. The Cooperative Institutional 
Research Program (CIRP) Annual Freshman Survey provides 
a very useful assessment tool for understanding what an 
institution's students are like when they begin their first year 
of college. The survey is administered to first-year students 
during either an orientation session before the start of or early 
in the fall semester. On this survey, a wide range of student 
characteristics are assessed, such as values and attitudes, high 
school experiences, goals and aspirations, and educational and 
career plans. 



Campus Presentations of CIRP Data 



Several campus groups typically find CIRP data useful 
for strategic planning. For instance, a discussion of student 
characteristics is conducted with faculty and staff who teach 
first-year seminar courses. Instructors are given an overview 
of the personal and professional goals of new students and 
their expectations for achievement in college. Results are also 
presented at special academic programs (such as graduate 
teaching assistant training programs) and to the student af- 
fairs staff. Consequently, these assessment findings have been 
used to support several types of campus initiatives for first-year 
students. 

Another campus group to whom CIRP data are annually 
presented is the admissions/ enrollment committee. Here, the 
discussion focuses on the reasons students are going to col- 
lege and their reasons for selecting this particular college or 
university. CIRP results provide meaningful data regarding 
specific facets of the institution that are important to students 
when making their college choice. Factors such as why they are 
going to college (e.g., personal goals, career goals, and family 
influences), reasons for selecting this institution (e.g., location, 
tuition, academics, and special programs), and family informa- 
tion (e.g., parental income and education) are considered. In 
addition, these data may be further broken down by students' 
major field to enable university planners to monitor student 
characteristics and develop programs to improve students' 
first-year experience across the disciplines. 
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Student Characteristics and First- Year Outcomes 

An important part of assessing first-year students' characteristics is conducting lon- 
gitudinal studies to determine how those initial characteristics are related to subsequent 
college outcomes. For instance, an important indicator of student progress is retention 
to the second year of college. Northern Illinois University has conducted several studies 
and found that student characteristics assessed by the CIRP are significant predictors of 
students' persistence to their second year. Individual items on the survey can be grouped 
together to comprise several factors, and those factors are related to retention. An assess- 
ment of almost 10,000 students found that high school curriculum, academic self-concept, 
and achievement expectancies were all significant predictors of college persistence for two 
years (House, 1996). 

Another critical measure of student success during the first year of college is achieve- 
ment in science and mathematics courses. Because many careers such as business, health 
sciences, and engineering require mathematical skills, lower achievement levels during 
the first year of college can restrict students' choices of majors and affect their subsequent 
career paths. Several assessments have found CIRP data to be helpful for understanding 
factors that influence student success in science and mathematics. For instance, student 
self-ratings of their mathematical ability and overall academic ability were significant pre- 
dictors of achievement in a finite mathematics course taken during the first year (House, 
1995b). Similarly, the same variables measured by the CIRP Annual Freshman Survey were 
significant predictors of achievement in first-year college chemistry (House, 1995a). More 
recently, we have found that several student characteristics measured by the CIRP (such 
as achievement expectancies, academic self-concept, and financial goals) were predictors 
of the first-year grade performance of students in science, engineering, and mathematics 
(House, 2000). 



Linking CIRP Data With Other Information Sources 

In order to facilitate student growth during college, several types of instructional 
practices are used and various out-of-class experiences are typically provided. However, 
the effects of specific instructional strategies and student experiences on college outcomes 
still need to be assessed. One of the attractive features of the CIRP Annual Freshman Sur- 
vey is the ability to merge student data with assessment data obtained from other surveys. 
For instance, the Your First College Year (YFCY) survey is designed to assess a number of 
the same dimensions as the Freshman Survey, as well as academic and social experiences 
during the first year of college. In addition, the College Student Survey (CSS) measures 
more than 200 college experiences, student changes, and outcomes. Individual student 
responses on these surveys can be merged into a single large data set. These surveys can 
be used to provide a longitudinal assessment of students at the time they began college, at 
the end of their first year, and at a later point in their college career. This strategy allows 
an assessment of student growth and achievement and an examination of college experi- 
ences that contribute to that growth, thereby providing a unique perspective on student 
development. 

Conclusion 

In many instances, institutions use their CIRP data to provide a cursory overview of 
the characteristics of their new first-year students. However, it is important to realize that 
the arrival of the CIRP data mark the beginning of an effective use of the data. Institutional 
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improvement can be realized by the data being prepared and provided to the appropriate 
campus offices. The institutional researcher becomes a key campus resource by conduct- 
ing campus-based research using CIRP data and by linking CIRP data with other campus 
data sources. This process allows CIRP data to be incorporated into a more broad-based 
assessment program directed toward understanding students' first-year experiences and 
outcomes. There needs to be consistent communication between institutional research- 
ers, first-year experience coordinators, and campus CIRP representatives. Further, CIRP 
data can provide an important foundation for comprehensive assessment efforts and be a 
strategic resource for improving first-year programs for students. 

References 

House, J. D. (1995a). Noncognitive predictors of achievement in introductory college 
chemistry. Research in Higher Education, 36, 473-490. 

House, J. D. (1995b). Noncognitive predictors of achievement in introductory college 
mathematics. Journal of College Student Development, 36, 171-181. 

House, J. D. (1996). College persistence and grade outcomes: Noncognitive variables as pre- 
dictors for African-American, Asian- American, Hispanic, Native American, and White students. 
Paper presented at the Association for Institutional Research Annual Forum, Albuquerque, 
NM. (ERIC Document Reproduction Service No. ED 397 710) 

House, J. D. (2000). Academic background and self-beliefs as predictors of student 
grade performance in science, engineering, and mathematics. International Journal of In- 
structional Media, 27, 207-220. 



What Students Expect May Not Be What They Get: The PEEK 

(Perceptions, Expectations, Emotions and Knowledge about 
College) 




Claire Ellen Weinstein, 



"I can't wait to go to college. . .Classes meet only a few hours 

’ each day and the rest of the time is mine!!" 

Cynthia A. King, 3 

Pe SSy Pri Hsuan Hsieh, „j didn't spend a lot of time studying in high school, but 

Taylor W. Acee, m y g rac j es were okay so I shouldn't have to work very hard in 
& David R. Palmer co n e ge." 

"Hey. . .if I get in trouble in my college courses, my teachers 
will tell me exactly what to do." 



As we can see from these three quotes, many entering col- 
lege students have unrealistic or incorrect perceptions, expecta- 
tions, emotions, or knowledge about what college life will be 
like for them. Researchers in higher education (e.g., Astin, Tinto, 
and Pascarella) have often pointed out the danger of students' 
misperceptions and unrealistic expectations on their academic 
achievement, satisfaction, and retention, particularly in students' 
first year of college. Success in school is not simply a result of 
possessing good reading, math, writing, and study skills. Stu- 
dents' expectations about their college experience have a pow- 
erful influence on their thoughts and actions when they get to 
college. The degree to which students' expectations accurately 
reflect their college environment will have a critical impact on 
their academic performance and satisfaction. Many academi- 
cally able and gifted students drop out of college during their 
first year because of personal, social, or academic expectations 
that are inaccurate or not fulfilled. 



What is the PEEK? 

The Perceptions, Expectations, Emotions and Knowledge 
(PEEK) about College (Weinstein, Palmer, & Hanson, 1995) is 
an assessment tool designed to help identify students' ideas, 
attitudes, beliefs, and expectations about what college will be 
like for them. The PEEK is a brief screening measure designed 
to help students as well as student affairs specialists, advisors, 
counselors, and bridge program or first-year seminar instructors 
identify possible discrepancies between students' beliefs, expec- 
tations, and knowledge about college and the realities they will 
face at a particular institution. Once any discrepancies have been 
identified, more accurate and realistic perceptions and expecta- 
tions can be developed by a variety of interventions in settings 
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such as orientation programs, advising sessions, bridge programs, first-year seminars, 
and freshmen interest groups (FIGS) or learning communities. 

The PEEK is a self-report assessment consisting of 30 items, which are organized 
into three categories with 10 items in each category. A student's perceptions, expectations, 
emotions, and knowledge about college fall into three general categories of the college 
experience: (a) personal, (b) social, and (c) academic. Students are asked to indicate, 
using a five-point rating scale ranging from "not at all likely" to "extremely likely," the 
extent to which they expect to have a particular experience in college. Although norms 
created by a national team of experts are available, colleges need to consider what is 
realistic for their institution. What is "normal" for one college setting may not be nor- 
mal for another one. For example, expecting a lot of guidance on how to succeed in 
college may be appropriate for a small college setting but not for large state colleges or 
universities. Also, expectations about maintaining close contact with family and friends 
is more likely to occur at a local college campus than if the student moves to another city 
or state to attend college. Finally, expecting individual help from an instructor is more 
likely to occur in a relatively small community college course than in a lower-division 
course at a major research university with an enrollment of 100 or more students. For 
these reasons, it is recommended that colleges establish their own norms to compare 
students' responses to the realities of their own institutions. 

The PEEK Categories 

The PEEK yields information about the personal, social, and academic dimensions 
of the college experience. 

Personal dimension. Items in the personal category measure students' expectations 
about their emotional reactions to college, the degree to which they are prepared to do 
college-level work, how college fits into their future goals, and the degree to which they 
will take personal responsibility for their own learning. A sample item is: "I think it 
will be harder to succeed in college than it was in high school." 

Social dimension. Items in the social category measure students' expectations 
about social pressures in college; interactions with instructors; the nature and make-up 
of college populations; and their relationships with family, students, and friends. A 
sample item is: "I will have a lot of free time for non-academic activities while I am in 
college." 

Academic dimension. Items in the academic category measure students' expecta- 
tions and knowledge about the difficulty of college courses, the nature of learning in 
college, the roles and responsibilities of college instructors, and the nature of instruc- 
tion in college. A sample item is: "If I am having difficulty in a course, the instructor 
will tell me." 



Scoring the PEEK 

The PEEK can be hand- or machine-scored. Machine scoring can be done using software 
provided to users. It can also be scored by the publisher using a default system created by 
the authors. This system is based on national data from students and higher education ex- 
perts. It is also easy to create custom scoring based on the realities of individual institutions. 
Because what is a reasonable and realistic expectation for one school may not be appropri- 
ate for another, users might want to consider tailoring the scoring to their own institution. 
Again, these adjusted scoring rubrics can be analyzed locally or through the publisher. 
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Development of the PEEK 

Exploratory data were collected from more than 3,000 college faculty and students at 
all levels of higher education over a period of about three to four years. This information 
about possible differences in the personal, social, and academic environments of different 
colleges was used to develop a large database of items (more than 300 potential items were 
created). More than two years were spent testing and refining this database to produce a 
usable (brief, but informative) measure that can help identify students' perceptions and 
expectations so that these expectations can then be compared to the likely realities of their 
college environment as determined by the institution. 

Uses for the PEEK 

For each student, PEEK data provide the baseline information needed to help students 
adjust successfully to new situations and circumstances. Typically, the PEEK is administered 
either prior to students" arrival on campus (e.g., summer bridge programs or orientation) 
or shortly after they begin taking classes (e.g., as part of advising or a first-year seminar). 
Some ways institutions can use the PEEK, include: 

1. Increasing student awareness. Examining, confronting, clarifying, and reconsid- 
ering their own perceptions will help students become more aware of their 
expectations about college. This increased awareness can alert students to 
possible problem areas and encourage an exploration of any inconsistencies. 

The first step in correcting misperceptions is to know they exist. 

2. Counseling, advising, and orienting students. Students whose perceptions and 
expectations appear to be somewhat inconsistent with those of the institution 
are likely to benefit from interventions that explore those inconsistencies. 

3. Supplementing instruction in a first-year seminar or program. Individual PEEK 
results can be used in first-year seminars or programs to generate valuable 
discussions aimed at developing more realistic ideas of what attending college 
will be like for students. Depending on individual PEEK responses, the instruc- 
tor can adapt the curriculum to meet the needs of the class or individualize 
the curriculum through activities and small-group work. 

We know from the findings of many researchers that it takes more than good study 
skills and learning strategies to succeed in college. The earlier we identify students' erro- 
neous perceptions and expectations about college life, the greater our chances of helping 
them adjust to the opportunities and demands of college. 
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Assessing Student Expectations of College: 

The College Student Expectations Questionnaire 

Robert M. Gonyea What do new students think their first year of college 

will be like? In what learning activities do they anticipate en- 
gaging? What do they want from their institution to achieve 
their educational and social goals? Among the most influen- 
tial educational statements ever circulated in American higher 
education is the "Principles of Good Practice in Undergraduate 
Education" (Chickering & Gamson, 1987). Its sixth principle 
is that "good practice communicates high expectations." The 
call for high expectations applies to both faculty members 
and students; the former must hold their students to high 
performance standards and students must expect more from 
themselves and their teachers. 

An expectation is more than a wish or a hope. Rather, it 
is something a student imagines will happen, anticipates doing 
or experiencing, or even requires from the institution. Expec- 
tations are grounded in a student's self-understanding and in 
knowledge about the institution that he or she plans to attend. 
From the perspective of the individual student, an expectation 
is like a plan or a goal. When directed at the institution, it may 
represent the conditions by which the student will measure 
his or her satisfaction with the institution and college life in 
general. As such, expectations of the institution are a form of 
psychological contract (Rousseau, 1995). 

Expectations affect the college experience in at least two 
ways. The first is to act as an organizing system or filtering 
mechanism that helps the individual student determine what 
is or is not worth attending to or putting effort toward. That is, 
expectations influence choices, behaviors, and experiences so 
as to construct what becomes reality for the individual (Feld- 
man, 1981). The second is to act as a stimulus or deterrent to 
behavior, as represented by psychological theories such as ex- 
pectancy theory, self-efficacy theory, and motivational theories 
(Kuh, 1999; Olson et ak, 1998). Kuh (2000) writes "to maximize 
learning and involvement during the first year of college, stu- 
dents need to set personal goals that are high enough so that 
they must try their best in classes and use campus resources to 
augment what they are learning in their classes." In addition, 
when a student's expectations are met, he or she is more likely 
to remain in school and complete a degree. When expectations 
are unmet, the student may consider dropping out or trans- 
ferring to an institution with a better fit (Braxton, Hossler, & 
Vesper, 1995). 
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The College Students Expectations Questionnaire 

The College Student Expectations Questionnaire (CSXQ) is a shortened version 
of its parent instrument, the College Student Experiences Questionnaire (CSEQ), both 
are authored by Kuh and Pace. The CSXQ was originally developed in 1997 for use in a 
project supported by a Fund for the Improvement of Post Secondary Education (FIPSE) 
grant directed by Karen Schilling at Miami University. The CSXQ measures beliefs about 
how students will spend their time during the first year. When paired with the CSEQ, 
administered as a follow-up measure toward the end of the first year, the institution can 
assess the degree to which those expectations were met. 

To date, the CSXQ has been completed by more than 50,000 students at more than 
75 different colleges and universities. It is typically administered to new students before 
the start of classes, at orientation, or during welcome week. The paper instrument is 
four pages in length and takes about 10 minutes to complete, and an online version is 
also available. The CSEQ Research Program processes and scores the survey under the 
Indiana University Center for Postsecondary Research. Participating institutions receive 
a floppy disk or CD containing their raw data, an electronic copy of the results tables in 
SPSS format, and a printed copy of the output. For institutions that administer the CSEQ 
as a follow-up at the end of the first year, a special report comparing CSXQ and CSEQ 
data can be produced along with the basic CSEQ report. 

The CSXQ shares 87 items in common with the CSEQ, not including questions about 
background characteristics. The common items include expectations of college activities 
and of the campus environment. The college activities items on the CSXQ are presented 
in 11 categories: 

• Library and information technology 

• Student interactions with faculty members 

• Course learning activities 

• Writing experiences 

• Campus programs and facilities 

• Clubs and organizations 

• Student acquaintances 

• Scientific and quantitative experiences 

• Topics of conversation 

• Information in conversations 

• Amount of reading and writing 

The campus environment items cover student expectations for the emphases placed 
on scholarly /intellectual qualities and expectations about the personal/ social climate of 
the campus. This section also asks students what they suppose relationships will be like 
with other students, faculty members, and administrative personnel and offices. 

Making the Most of CSXQ Data 

Discovering what students expect from their college experience is crucial if faculty 
members are to adjust their instructional approaches accordingly and institutions are to 
modify policies and practices to respond in educationally effective ways. Below are three 
examples of how CSXQ data can be used to improve the first-year experience: 
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1. Expectations data can identify areas where socialization efforts are needed 
to help students develop appropriate attitudes and behaviors to succeed in 
college. CSXQ information can be used to tailor new student recruitment 
materials and orientation activities where it is discovered that student ex- 
pectations need to be modified in order for students to succeed. It is also 
possible to provide formative feedback to individual students based on their 
CSXQ responses. The CSEQ Research Program makes the Student Advising 
Report available for this purpose. 

2. Expectations data can help faculty members, administrators, and student af- 
fairs professionals better understand who new students are and what they 
want from their college experience. Faculty members who teach traditional 
first-year courses, especially those who teach first-year experience courses, can 
use CSXQ data to influence student expectations directly. Academic advisors, 
residence hall staff, and campus activities staff can implement programs to 
encourage students to expect more and set higher learning goals. Both faculty 
and staff members should take it upon themselves to understand student 
expectations so that they can fashion more suitable learning environments 
for first-year students. 

3. Expectations data can identify areas where improvements in teaching and 
learning may be needed and complement initiatives, such as the scholarship 
of teaching and learning. Typically, the data show that in most areas students 
are more optimistic for their first year than they subsequently realize. This is 
commonly known as the "freshman myth." For example, they study fewer 
hours, write less, and interact with faculty members less than they expect 
to. This pattern of results prompts questions about whether the nature and 
amount of assigned academic work is appropriate to cultivate the range and 
depth of intellectual skills required to succeed in college and beyond. 

The CSXQ may also function as an instructional device to help students navigate 
through their first year of college. While filling out the questionnaire, new students 
pause to contemplate the learning opportunities that will be presented to them during 
the coming school year. This leads them to formulate or rethink expectations in areas 
where they have not given much thought. 
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How often do your first-year students ask their profes- 
sors for advice to improve their writing? To what extent does 
your institution emphasize an appreciation for diversity and 
ideas? What gains have your first-year students made in their 
ability to synthesize ideas, think analytically, write effectively, 
get along with others, and understand new technology? And 
how often do your students have serious discussions with 
others whose background and political opinions were differ- 
ent from theirs? 

The CSEQ is useful for assessing the first-year experience 
of students and helping institutions identify college activities 
and environmental factors that foster student development 
and learning outcomes. Examples include how frequently 
first-year students ask their professors for advice to improve 
their writing; the extent to which the institution emphasizes 
an appreciation for diversity and ideas; the overall gains first- 
year students have made in their ability to synthesize ideas 
think analytically, write effectively, get along with others, and 
understand new technology; and how frequently students take 
part in serious discussions with others whose background 
and political opinions are different from their own. CSEQ 
is unique among assessment instruments in that it measures 
the "quality of effort" students expend in three aspects of the 
collegiate experience: 



1. Campus resources 

2. Aspects of the college environment that foster 
learning and encourage students to take advan- 
tage of campus resources 

3. The progress — or, self-reported gains — students 
make in desirable learning outcomes 

Among the examples of its usefulness in first-year 
student assessment, the CSEQ can provide a comparison of 
the experiences of students involved in Freshman Interest 
Groups, or FIGs, with a group of students not involved in 
FIGs. It can provide a measure of satisfaction among first- 
year students concerning the quality of their relationships 
with faculty and staff. And the instrument can help institu- 
tions identify patterns of student use of the library and other 
campus facilities. 
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What is the CSEQ? 

Developed in the late 1970s as a national survey tool by then-UCLA professor C. Rob- 
ert Pace, the CSEQ has been administered at more than 400 colleges and universities and 
completed by more than 300,000 students. The CSEQ has been well-received by the higher 
education community — a 1994 National Center for Education Statistics report cited the 
CSEQ for its "excellent psychometric properties" (Kuh & Siegel, 2000). Now in its fourth 
edition, the CSEQ has produced five sets of national norms data. In addition, the CSEQ has 
been cited in more than 250 articles, books, and dissertations, and hundreds of institutional 
reports (Gonyea, Kish, Kuh, Muthiah, & Thomas, 2003). 

The CSEQ is an eight-page questionnaire with more than 160 items that ask students 
to report on the activities in which they are engaged during the current academic year. In- 
stitutions also have the option of administering the CSEQ online via the World Wide Web, 
in a convenient, paperless format. The web version is identical to the paper instrument. 
Because the CSEQ asks students to reflect on their experiences during the current school 
year, institutions administering the instrument to first-year students do so in the spring of 
the academic year (typically between March and May, or whenever the school's academic 
calendar is between two-thirds and three-fourths completed). It can also be administered 
in the fall to sophomores, at which time students are asked to reflect on their first-year 
experiences. The CSEQ offers a high degree of flexibility in terms of sampling, allowing 
institutional users to select populations and sampling methods and encouraging them to 
develop strategies for administration that best meet their local assessment needs. 

What Does the CSEQ Measure? 

A versatile and comprehensive self-report assessment tool, the CSEQ encourages 
students to engage in reflection and self-evaluation. As a result, the instrument provides 
institutions with several valuable pieces of information about their students. In addition 
to providing student background characteristics, it measures the extent to which students 
engage in activities known to be important to their learning and personal development. It 
also determines the level of satisfaction students have with their institution and provides 
student ratings of key characteristics of the college environment. Finally, the instrument 
reports estimates of student gains — or progress — toward important learning objectives. 

What are the Major Components of the Instrument? 

The CSEQ is divided into three major categories: (a) the Activities Scales, which mea- 
sure how students spend their time and use their institution's educational resources; (b) 
the Environment Scales, which measure students' perceptions of the college environment; 
and (c) the Gains Scales, which measure students' estimated progress in areas that experts 
consider to be related to desirable learning outcomes. 

The Activities Scales 

The Activities section of the instrument includes 13 scales that measure the quality of 
effort (time and energy) students expend in using institutional resources that are provided 
for their learning and development. 



1. Library experiences 
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2. Computer and information technology 

3. Course learning 

4. Writing experiences 

5. Experiences with faculty 

6. Art, music, and theater 

7. Campus facilities 

8. Clubs and organizations 

9. Personal experiences 

10. Student acquaintances 

11 . Science and quantitative 

12. Topics of conversation 

13. Information in conversations 

The College Environment Scales 

The CSEQ contains 10 scales related to students' perceptions of the college environ- 
ment. That is, the scales measure the extent to which colleges and universities foster rela- 
tionships that are supportive of learning and encourage students to engage in educationally 
purposeful activities. 

Seven of the rating scales refer to the nature by which the college environment em- 
phasizes various aspects of student learning and personal development, including: 

1. Academic, scholarly, and intellectual abilities 

2. Aesthetic, expressive, and creative abilities 

3. Critical, evaluative, and analytical abilities 

4. An understanding and appreciation of human diversity 

5. Information literacy skills 

6. Vocational and occupational competence 

7. Personal relevance and practical value of courses 

Three of the environment rating scales refer to the nature of students' relationships 
with others, specifically: 

1. Other students and student groups 

2. Faculty members 

3. Administrative personnel and offices 

Estimate of Gains Scales 

The Estimate of Gains scales consist of student ratings of their progress toward im- 
portant educational goals and outcomes. There are 23 gains items on the instrument that 
are organized in five major clusters: 

1 . General education, literature, arts, and social sciences 

2. Personal development and social competence 

3. Science and technology 

4. Intellectual skills 

5. Vocational competence 
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The CSEQ provides space for 20 additional questions that institutions can use for locally 
developed questions related to assessment of specific first-year programs and initiatives. 
The additional questions section can also be used as a coding device so that groups can 
be identified and compared for further analysis. For instance, if an institution wanted to 
compare the experiences of students in theme housing with those in non-theme housing, 
the additional questions section might be used to inquire about specific experiences of the 
two groups. Similarly, the two groups might be coded in the answer spaces provided for 
question 3, for instance, as "3- A" and "3-B," respectively. 

How is Institutional Data Returned? 

Institutional users receive a computer diskette that contains the institution's data, 
as well as a printed report of frequency counts and descriptive statistics. Additional data 
analyses such as t-tests, ANOVAs, correlations, and sub-group analysis can be performed 
upon request. Institutions can also request a customized comparison of their data with peer 
institutions for benchmarking purposes. 

In closing, research has indicated that quality of effort is one of the best predic- 
tors of the effects of attending college (Ewell & Jones, 1996; Kuh, Pace, & Vesper, 1997). 
Because it provides an estimate of the contributions students make to their own learn- 
ing and measures the extent to which students use the resources institutions offer, 
the instrument is a highly effective tool for helping institutions understand student 
involvement in, and satisfaction with, college. In sum, the CSEQ can be a highly use- 
ful proxy for the value that college students place on their experience and a yardstick 
by which student involvement in educationally purposeful activities can be measured. 
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ollege Student 
uestionnaire (CCSEQ) 

The Community College Student Experience Question- 
naire (CCSEQ) (Friedlander, Pace, Murrell, & Lehman, 1990, 
revised 1999) is a standardized self-report survey instrument 
that examines the degree of interaction between first-year 
students and the community college. It gathers information 
in four areas: 



• Amount, breadth, and quality of effort in both 
in-class and out-of-class experience 

• Progress toward important educational outcomes 

• Satisfaction with the community college environment 

• Student demographic and background characteristics 

The instrument is based on the concept that what contrib- 
utes most to students' progress toward important outcomes of 
college is not who they are or what they are, but what they do 
while they are there (Pace, 1984). 

At the heart of the CCSEQ is the College Activities sec- 
tion, designed to measure student quality of effort in the use 
of facilities provided by the institution and experiences with 
other students, faculty, and staff members at the community 
college. Respondents indicate the number of times during the 
current school year they participated in activities that stimulate 
the development of academic skills and social growth in the 
following areas: 

• Contact with faculty members, other students, 
organizations, counseling and career centers 

• Experiences with writing 

• Engagement in classroom activities 

• Instruction in study and vocational skills 

The 1999 revision incorporates language about the new uses 
of technology and also adds a new scale devoted specifically to 
assessing the students' quality of effort in using computer tech- 
nology. 

The items comprising the Quality of Effort Scales require 
incrementally greater effort; items at the end of the list subsume 
items at the beginning. Scale scores are computed by summing 
the scores for items in a group. The higher the score on a scale, the 
greater the degree of involvement on that scale. The scales measure 
the following categories of community college participation: 
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• Course activities 

• Library activities 

• Faculty-student interaction 

• Computer technology 

• Student acquaintances 

• Art / music / theater activities 

• Writing activities 

• Science activities 

• Career/ counseling activities 

The Estimate of Gains section of the CCSEQ measures students' self-reported progress 
in six areas: 

• Career preparation 

• Arts 

• Communication skills 

• Mathematics and science 

• Technology 

• Personal and social development 

• Perspectives of the world 

Lehman (1991) found strong relationships between several Quality of Effort Scales 
and Estimate of Gains items, leading her to conclude, "The more effort community college 
students put into their college experiences, the more likely they are to make personal and 
academic gains" (p. 19). 

Eight questions elicit information about students' perceptions about the nature of their 
college environment, forming a College Environment scale. These items may be used to 
form a satisfaction scale to indicate how supportive, helpful, and challenging students find 
the community college environment. 

Results of the CCSEQ are provided to participating community colleges in the form of 
written reports and on diskette. These include frequency counts and descriptive statistics. 
Additional analyses are available upon request. 

In addition, the Test Manual and Comparative Data (Ethington, Cuthrie, & Lehman, 
2000), published by the Center for the Study of Higher Education, contains information 
gathered from over more than 19,000 students who have completed the 1999 revision of the 
CCSEQ at 40 community colleges. More than 65,000 students at 136 community colleges 
have completed the instrument since its initial publication in 1990, providing national data 
for benchmarking. 

Information provided by the CCSEQ gives community college administrators and 
faculty a blueprint for operationalizing theoretical concepts of student involvement and 
engagement. It places responsibility for the learning on students and holds them account- 
able for their use of the programs and facilities provided by the institution. It provides 
valuable information about the interactive processes between students and the institution 
that is vital if we are to put together the puzzle of institutional impact to enhance the social, 
academic, and career development of community college students. 
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Knowing How to Learn is as Important as Knowing What to Learn: 
Th&Learning and Study Strategies Inventory 



The Learning and Study Strategies Inventory (LASSI, 2nd 
edition) is based on both cognitive and metacognitive conceptions 
and models for understanding how we learn, think, and thrive in 
academic environments, in particular, and learning environments, 
in general. Cognitive and metacognitive control strategies are all 
based on three general stages: (a) awareness, (b) reflection, and 
(c) action/ control. In order to be more effective and efficient in 
achieving academic success, college students need to become 
aware of what they do (or fail to do) that helps them study and 
learn in college. However, many students have little or no idea 
about their strengths and weaknesses as learners. If students do 
not think they need help succeeding in their studies, they may find 
it difficult to benefit from a first-year seminar or college learning 
course. Students must become aware of what they do, think, and 
believe when they are studying and learning and reflect on where 
they may need to change or improve. The LASSI helps students 
gain the awareness they need to benefit from interventions or 
courses designed to help them become more strategic, self-regu- 
lated learners. LASSI results can also be used by instructors and 
student affairs specialists to target interventions to the particular 
needs of their students. 



What is the LASSI Assessment and What Does it Measure? 

The LASSI is a 10-scale, 80-item self-report assessment of 
students' awareness about and use of learning and study strategies 
related to skill, will, and self-regulation components of strategic 
learning. Students respond to each item using a five-point Likert- 
type scale ranging from "not at all typical of me" to "very much 
typical of me." The focus is on both covert and overt thoughts, 
behaviors, attitudes, and beliefs that relate to successful learning 
and that can be altered through educational interventions. Re- 
search has repeatedly demonstrated that these factors contribute 
significantly to success in college and that they can be learned or 
enhanced through educational interventions such as a first-year 
seminar or learning and study skills courses. It can be adminis- 
tered in either an online or a paper-and-pencil format. 

The LASSI is designed to simplify administration and scoring 
as much as possible without losing power or diagnostic informa- 
tion. To help achieve this goal, it uses a self-report format and 
does not require any special administration procedures, such as 
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specially trained personnel. The directions are included on the front of each printed booklet 
so it can be used in both individual and group settings. The web-administered version pres- 
ents the same directions to each user. The LASSI is not a timed measure, but most students 
complete it in approximately 15 to 20 minutes. Self-scoring the print version takes approxi- 
mately 10 minutes. 

The LASSI provides standardized scores (percentile score equivalents) and national 
norms for 10 different scales (there is no total score because this is a diagnostic measure). 
The LASSI is both diagnostic and prescriptive. It provides students with a diagnosis of their 
strengths and weaknesses, compared to other college students, in the areas covered by the 10 
scales. It is prescriptive in that it provides feedback about areas where students may be weak 
and need to improve or change their knowledge, attitudes, beliefs, and skills. 

Components of Strategic Learning 

The Skill Component. The LASSI scales related to the skill component of strategic learning 
are: (a) Information Processing, (b) Selecting Main Ideas, and (c) Test Strategies. These scales 
examine students' learning strategies, skills, and thought processes related to identifying, 
acquiring, and constructing meaning for important new information, ideas, and procedures. 
The scales also examine how students prepare for and demonstrate their new knowledge on 
tests or other evaluative procedures. 

The Will Component. The LASSI Scales related to the will component of strategic learning 
are: (a) Attitude, (b) Motivation, and (c) Anxiety. These scales measure students' receptivity to 
learning new information, their attitudes and interest in college, their diligence, self-discipline, 
willingness to exert the effort necessary to successfully complete academic requirements, and 
the degree to which they worry about their academic performance. 

The Self-Regulation Component. The LASSI Scales related to the self-regulation compo- 
nent of strategic learning are: (a) Concentration, (b) Time Management, (c) Self-testing, and 
(d) Study Aids. These scales measure how students manage, or self-regulate and control, 
the learning process by using their time effectively; focusing their attention and maintaining 
their concentration over time; checking to see if they have met the learning demands for a 
class, an assignment, or a test; and using study supports such as review sessions, tutors, or 
special features of a textbook. 

The LASSI Scales 

The Anxiety Scale assesses the degree to which students worry about school and their 
academic performance (sample item: "I feel very panicky when I take an important test."). 
Students who score low on this scale are experiencing high levels of anxiety associated with 
school (note that this scale is reverse scored). High levels of anxiety can direct attention 
away from completing academic tasks. Students who score low on this scale may need to 
develop techniques for coping with anxiety and reducing worry so that their attention can 
be focused on the task at hand. 

The Attitude Scale assesses students' attitudes and interest in college and academic suc- 
cess. It examines the degree to which their approach to college and academics is facilitative 
or debilitative (sample item: "I only study the subjects I like."). Students who score low on 
this scale may not believe college is relevant or important to them and may need to develop 
a better understanding of how college and their academic performance relates to their future 
life goals. 

The Concentration Scale assesses students' ability to direct and maintain attention on 
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academic tasks (sample item: "If I get distracted during class, I am able to refocus my atten- 
tion."). Low scoring students may need to learn to monitor their level of concentration and 
develop techniques to redirect attention and eliminate interfering thoughts or feelings so that 
they can be more effective and efficient learners. 

The Information Processing Scale assesses how well students' can use imagery, verbal 
elaboration, organization strategies, and reasoning skills as learning strategies to help build 
bridges between what they already know and what they are trying to learn and remember 
(sample item: "I try to find relationships between what I am learning and what I already 
know."). Students who score low on this scale may have difficulty making information mean- 
ingful and recalling it in the future. 

The Motivation Scale assesses students' diligence, self-discipline, and willingness to exert 
the effort necessary to successfully complete academic requirements (sample item: "When 
work is difficult, I either give up or study only the easy parts."). Students who score low on 
this scale need to accept more responsibility for their academic outcomes and learn how to 
set and use goals to help accomplish specific tasks. 

The Selecting Main Ideas Scale assesses students' skill at identifying important informa- 
tion for further study from among less important information and supporting details (sample 
item: "When studying, I seem to get lost in the details and miss the important information."). 
Students who score low on this scale may need to develop their skills at identifying impor- 
tant information on which to focus their attention. Tasks such as reading a textbook can be 
overwhelming if students focus on every detail presented. 

The Self-Testing Scale assesses students' use of reviewing and comprehension monitor- 
ing techniques to determine their level of understanding of the information to be learned 
(sample item: "I stop periodically while reading and mentally go over or review what was 
said."). Low-scoring students may need to develop an understanding of the importance of 
self-testing and learn effective techniques for reviewing information, monitoring their level 
of understanding, or applying what they are learning. 

The Study Aids Scale assesses students' use of supports or resources to help them learn 
or retain information (sample item: "I try to find a study partner or study group for each of 
my classes."). Students with low scores may need to develop a better understanding of the 
resources available to them and how to use these resources to help them be more effective 
and efficient learners. 

The Test Strategies Scale assesses students' use of test-preparation and test-taking strategies 
(sample item: "I review my answers on essay tests to make sure I have made and supported 
my main points."). Low-scoring students may need to learn more effective techniques for 
preparing for and taking tests so that they are able to demonstrate their knowledge of the 
subject matter effectively. 

The Time Management Scale assesses students' application of time-management principles 
to academic situations (sample item: "I find it hard to stick to a study schedule."). Students 
who score low on this scale may need to develop effective scheduling and monitoring tech- 
niques in order to assure timely completion of academic tasks and to avoid procrastination 
while realistically including non-academic activities in their schedule. 

While the LASSI helps with awareness of students' strengths and weaknesses as a learner, 
the reflection and action/ control stages of cognitive and metacognitive models need to be 
addressed by instructional interventions, such as workshops, a first-year seminar or a learning- 
to-learn course, or the LASSI Instructional Modules. For example, many colleges administer 
the LASSI at the beginning of first-year seminars so that instruction can be targeted to stu- 
dents' needs. The LASSI Instructional Modules are web-based instructional units containing 
approximately three to five hours of instruction in each of the 10 areas assessed by the LASSI. 
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These modules can be used independently by students, or they can be incorporated into a 
learning to learn course. For example, we use the modules in our learning-to-learn course so 
that students have more time for guided practice and feedback in class. 
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Students' experiences during their first few weeks on 
campus are highly influential. Integrating campus services and 
proactively reaching out to incoming students, while managing 
the expectations of college and university life, help incoming 
students to make the transition successfully 

Institutions with strong retention rates intervene early in the 
term to foster student success and retention. By front-loading the 
first term with the appropriate resources, it is possible to unlock 
the academic potential and the capacity for success that each 
student brings to campus. In far too many cases, students drop 
out before the institution is even aware of their needs. With the 
Retention Management System (RMS), institutions enact preven- 
tion plans for individual students early in the term. 



The Retention Management System 

The primary purpose of the Retention Management Sys- 
tem is to foster effective communication between students and 
their advisors, a purpose that is accomplished by identifying 
students' needs, attitudes, motivational patterns, resources, cop- 
ing mechanisms, and receptivity to intervention. More specifi- 
cally, this proactive approach to student retention is designed to 
enable institutions to: 



• Assess students' needs and strengths 

• Identify at-risk students 

• Recognize students' motivational patterns 

• Use information to implement successful referrals 
and intervention programs 

• Enable advisors to have effective and rewarding 
contact with students early in the first term 

The four basic components of the RMS that enable institu- 
tions to assess the motivations of incoming students are: 

• The College Student Inventory 

• The RMS Advisor /Counselor Report 

• The RMS Student Report 

• The RMS Summary and Planning Report 
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The College Student Inventory (CSI) 



The History 

The College Student Inventory is the foundation of the RMS and was designed especially 
for incoming first-year students. In 1981, Stratil, the author of the CSI, began research in the area 
of academic and social motivation with the goals of: 

• Creating a coherent framework for understanding human motivation in general 

• Identifying the specific motivational variables that are most closely related to 
persistence and academic success in college 

• Developing a reliable and valid instrument for measuring these variables 
(Stratil, 1988) 

As a result of his research, the original version of the CSI (titled the "Stratil Counseling In- 
ventory") was published in 1984. The current versions of the College Student Inventory — Form 
A and Form B — were published in 1988 and 2000 respectively, (see figure 1). 

Administering the CSI. Students complete the CSI as part of early orientation programs or 
during the first weeks of classes. Students complete Form A, a 194-item inventory that assesses 
a variety of motives and background information related to college success, in about an hour. 
Form B, a 100-item inventory, can be completed in approximately 30 minutes. Both are avail- 
able in the traditional paper-and-pencil format or online. Canadian and Spanish variations are 
offered as well. 

A number of scales are constructed from the inventory items to provide a detailed view 
of each student's motivation, coping ability, and receptivity to assistance. The main categories 
include: 



CSI-Form A 


CSI-Form B 


Academic Motivation 


Academic Motivation 


Social Motivation 


General Coping Ability 


General Coping Ability 


Receptivity to Support Services 


Receptivity to Support Services 


Internal Validity 


Initial Impressions 




Internal Validity 





Figure 1 . Major Scales for Forms A and B of College Student Inventory. 

The Scales 

The heart of the CSI rests with the independent motivational scales constructed for each 
of the categories above. The specific scales for Form A and Form B are listed in Figure 2. The 
Initial Impressions Scale, included in Form A, focuses on a student's first impressions of the 
institution. The Internal Validity scale assesses a student's carefulness in completing the inven- 
tory. This scale enables the institution to determine the care and attention the student gave to 
the test-taking. 



The RMS Advisor/Counselor Report 

This report provides information about the student's attitudes and motivations in 
percentile ranks. At a glance, advisors can identify the student's greatest areas of strength 
and needs. The most distinctive feature of this report (which does not appear on the 
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student's report) is the Summary of Academic Motivation, which provides a preliminary 
overview of a student's: 

• Dropout proneness 

• Predicted academic difficulty 

• Educational stress 

• Receptivity to institutional help 

In addition to the Summary of Academic Motivation, seven specific recommendations 
for each student are listed on this report, ranging from suggestions to "get help with writing 
skills" to "discuss emotional tensions with a counselor." The strength of each recommendation 
is indicated by its priority score. 

The motivational scales are reported in two ways: as a percentile rank and with a bar graph. 
With CSI Form A, background information about the student's high school academics, non-credit 
activities, family, and admissions test scores are also included in the profile. 



ACADEMIC MOTIVATION 


Form A 


Form B 


Study Habits 


Study Habits 


Intellectual Interests 


Intellectual Interests 


Academic Confidence 


Verbal Confidence 


Desire to Finish College 


Math Confidence 


Attitude Toward Educators 


Desire to Finish College 
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Figure 2. Components of major scales for Forms A and B of College Student Inventory. 
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The RMS Student Report 

The RMS Student Report is intended to give students insight into their own strengths 
and potential areas of need. It explains student's scores on each scale and is designed to 
encourage and guide. This report parallels the RMS Advisor/ Counselor Report but omits 
the Summary of Academic Motivation. 

The RMS Summary and Planning Report 

This report provides significant planning data and specific contact lists that name the 
students with needs in specific areas. Some key lists included in this report are: 

• Dropout proneness 

• Receptivity to institutional help 

• Need for academic assistance 

• Need for personal counseling 

• Need for career counseling 

• Need for social enhancement 

How are Institutional Data Returned? 

Online web links to the RMS Advisor /Counselor Report, the RMS Student Report, 
and the RMS Summary and Planning Report are provided to campus coordinators. Raw 
and translated data are also available online for institutions that wish to perform further 
analyses of their data. 



Conclusion 

Encircling your students with services during their first few weeks on campus may 
be the most effective strategy for enhancing student success and strengthening student 
retention. Hundreds of campuses nationwide find that early assessment of affective needs 
provides the foundation for timely and relevant intervention with their incoming class. 
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The Study Behavior Inventory (SBI) is a computerized, 46- 
item, self-report, diagnostic/ prescriptive survey that is the result 
of more than 10 years of research, development, and refinement. 
The instrument, which self-administers in approximately 15 min- 
utes, probes the study behaviors of college and university students 
and diagnoses behaviors that may result in their encountering 
academic difficulties so that students can receive appropriate 
remediation in these areas. After administering the SBI to more 
than 10,000 college and university students in a series of studies 
geared toward developing and standardizing the instrument, we 
believe we know a great deal about the connection between study 
behaviors and study skills. 

Distinguishing Between Study Behaviors and Study Skills 

Study behaviors are those actions students carry out when 
engaged in preparation for academic tasks. While the terms "study 
behaviors" and "study skills" have often been used synonymous- 
ly — and this has caused more than a little confusion — there is a 
clear distinction between the two concepts. Study skills represent the 
potential for action while study behaviors are the actions themselves. 
A student may have all the skills required — that is, he or she may 
be able to take good notes in class (possession of a study skill) — but 
may doodle in class (poor study behavior). This distinction is not 
trivial. Students are often placed in college and university devel- 
opmental education programs because high school grades, admis- 
sions tests, and placement tests suggest they do not use appropriate 
study behaviors (i.e., they do not take adequate notes; they do not 
use the library efficiently; they do not use study time well). We as- 
sume students do not exhibit study behaviors because they do not 
possess the appropriate study skills. Thus, these programs often 
focus on activities designed to develop skills such as note-taking 
skills, library-use strategies, and time-management ability. Yet, this 
assumption may be erroneous and unjustified. Often, these stu- 
dents do possess study skills but do not use them when preparing 
for academic work in college and university classrooms. 

The Underlying Concepts of the SBI 

The SBI is composed of 46 statements to which subjects 
respond on a four-point scale indicating how often a particular 
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statement might apply to them. Specifically, the response choices are (a) "rarely or never," 
(b) "sometimes," (c) "often or usually," or (d) "always or almost always." Student respon- 
dents are instructed to attempt to answer all items and to work until they have completed 
the instrument. Scores are coded so that positive responses (those indicating appropriate 
study behaviors) are coded high, while negative responses are coded low. 

A series of factor analytic studies consistently yielded three factors underlying study 
behaviors as measured by the SBI. This factor structure was found in samples from vary- 
ing locations in the United States, at community colleges and four-year institutions, and 
even in a Spanish-language edition (still under development) given to university students 
in four Latin American countries. Factor 1 deals with feelings of insecurity, efficacy as 
a student, and with levels of competitiveness in students when they approach academic 
tasks. Factors 2 and 3 include behaviors related to the appropriate use of time. Factor 2 
behaviors concern routine academic tasks such as doing assignments and preparing for 
classes. Factor 3 addresses behaviors dealing with long-range planning such as studying 
for examinations and the preparation of papers and other long-term projects. 

Factor 1: Academic Self-efficacy 

The connection between the belief about self and behavior is classic and well docu- 
mented. The Humanist, or Phenomenologist, psychologists of the 1960s and 1970s took 
this idea and made it a centerpiece of their therapeutic and educational strategies. Combs 
and Snygg (1959), note in their seminal book that, "What is perceived is not what exists, 
but what one believes exists... what we have learned to perceive as a result of our past 
opportunities and experiences" (pp. 84-85). Further, people's perceptions of themselves 
and their abilities are the determinants of their behavior (Patterson, 1973). It follows from 
this that students who perceive themselves as academically incompetent and helpless 
due to past academic experiences may determine that it is useless to try to excel academi- 
cally. We believe that academic self-efficacy, as measured in the first factor of the SBI, is 
the mediator between study skills and study behaviors. That is, a study skill will be used 
(i.e., will result in a study behavior) if students believe there is some chance they will be 
successful. An example of this is the student who knows all about using the library, its 
indexes, and catalogues (a study skill), but is so certain that any paper he or she writes 
will receive a failing grade, that he or she does not bother to use the library to complete 
a written assignment (a study behavior). Examples of SBI items that make up this factor 
include the following: 

1. I have to reread material several times; passages do not have much meaning 
the first time I go over them. 

2. I get nervous and confused when taking an examination and fail to answer 
questions to the best of my ability. 

3. During an examination, I forget names, dates, formulas, and other details I 
really do know. 

Factor 2: Time Management for Preparing for Routine, Short-term Academic Tasks 

This second factor underlying study behavior probes how students go about prepar- 
ing for routine, everyday tasks. Such tasks include reviewing class notes before the next 
class meeting, doing assigned readings between class meetings, and completing homework 
assignments. Items in this factor include: 
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1. I keep my assignments up to date by doing my work regularly from day to 
day. 

2. After a class lecture, I go back and recite to myself the material in my notes, 
rechecking any points I find doubtful. 

3. Before attending class, I prepare by reading or studying the assignments. 

Factor 3: Time Management for Preparing for Specific, Long-range Academic Tasks 

Finally, the third factor deals with behaviors that students exhibit when they are 
planning and carrying out specific, long-range academic tasks such as writing papers and 
studying for examinations. These activities involve planning over long periods of time and 
do not occur on a regular, day-to-day basis. Questions in this factor include: 

1. I find it hard to finish work by a certain time; work is unfinished, inferior, or 
not on time. 

2. When preparing for an examination, I learn facts in some logical order of 
importance, order of presentation in class or textbook, order of time in his- 
tory, etc. 



Scoring the SBI 

The SBI is fully computerized, completely networkable, and self-scoring. In a matter of 
seconds, respondents receive a hard copy of results that gives raw and percentile scores for the 
total instrument and for each of the three underlying factors. The package comes with a built- 
in set of recommended interventions, but user institutions can add their own specific sets of 
recommended interventions to the system by using the set-up programs that are included. 

Psychometric Properties 

The SBI was normed using a sample of more than 5,200 English-speaking college and 
university students in the United States. The standardization sample is representative by 
geographic area and type of institution (public and private, two-year and four-year). 

Validity. The factor structure obtained in the studies involving the development of the 
SBI is consistent with the educational and psychological theories upon which the instrument 
was based. (See the articles in the bibliography section for details of this theoretical base.) 
Additional evidence of the construct validity of the instrument is that the correlation between 
the total SBI score and the grade point average of survey respondents in the semester prior 
to their taking the inventory was .79. The correlations with Factors 1, 2 and 3 were .74, .70, 
and .67, respectively. Correlations with ACT on SAT total standard scores were .68 on the 
total score, and .65, .61, and .58 for the scores on the three factors. No differences between 
the scores of men and women were found on the total score or on any of the factor scores. 

Reliability. Cronbach's alpha for the scores on the three factors were .86, .82, and .70. 
These measures indicate high levels of internal consistency for the scores on each factor. 

Conclusion 

Currently, more than 300 institutions of higher education in the United States and in 
other English-speaking countries use the SBI. The instrument is a Windows-based package 
that includes a database management system, allowing scores of individual respondents 
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to be saved and statistically manipulated. The inventory can be loaded onto individual 
computers or networked throughout an institution. Unlike similar instruments, institu- 
tions receive a site license that permits them to install the SBI on as many computers as 
they wish or to network the inventory throughout the institution. There are no additional 
charges for individual administrations. Excellent technical support is available by e-mail 
or by telephone. Paper copies of the instrument are available for mass administrations that 
can then be scored using the computerized instrument. 
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For years, counselors have discussed the existence of a 
second curriculum in higher education — a curriculum that is 
essential for academic success. This is an affective curriculum — 
composed of habits, attitudes, and behaviors — that not only con- 
tributes to scholastic performance, but serves as its foundation. 
This foundation we have called academic aptitude. 

Research in the field of academic values (including our 
own) has identified about 25 possible factors related to academic 
aptitude. Factors in a student's life such as family involvement, 
time management, assertiveness, and responsibility have been 
shown to be important indicators of success. Each one of these 
had merit, but after 15 years of working with 140 colleges and 
20,000 students, eight areas of academic aptitude emerged, which 
we believe are the key foundational factors of college success. 
These eight factors are measured in the College Success Factors 
Index. 



Eight Factors of Academic Aptitude 

Responsibility/Control 

College success is often measured by a process requiring a 
gradual increase in responsibility year by year and an acceptance 
by the student that he has increasing control over his academic 
success. Understanding and meeting these responsibilities is an 
important predictor of success in college. Taking responsibility 
for one's success must be coupled with taking control. Know- 
ing expectations and taking the necessary control to meet them 
may be two different things, as are perceived effort and actual 
performance. 

Competition 

The need to compete is natural to most of us. Our society is 
individualistically oriented, rewarding those who excel beyond 
their peers. Getting an "A" on a test represents this phenomenon, 
as does further committing to and excelling in extracurricular 
activities. Astin (1984) found both content and resources ap- 
proaches to pedagogy tend to favor academically prepared, as- 
sertive students and that a passive approach to learning may be 
indicative of a lack of involvement. 
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Task Precision 

College success is partially defined by producing a series of class projects or 
tasks. Lower achievers lack persistence and conscientiousness in study skills, which 
may translate to reduced task precision. Some researchers feel the learning environ- 
ment must receive sufficient student effort and investment of energy to bring about 
the desired learning and development (Tinto, 1993). 

Expectations 

Success in school is also generated by one's own expectations to discover new and 
challenging experiences. This seeking is combined with setting one's own discovery 
course, instead of merely "taking in" what is offered by instructors. Successful students 
have their own goals and expectations related to assignments, areas of study, and fu- 
ture careers (Lunneborg & Lunneborg, 1976). Attaining goals and expectations and a 
relative posture towards knowledge development appears a necessary characteristic 
to success. 

Wellness 

Behavioral symptoms such as constant stress, sleeplessness, poor nutrition, or 
depression are often a deterrent to college success. Depression, anxiety, and stress have 
been known to impede school and college success; and burnout appears important as 
a factor in student attrition (Hallberg, Levitt, & Hallberg, 1984). 

Time Management 

From being on time to using one's time efficiently, time tends to be an ingredient 
in how successful students will be in college. Planning ahead and determining time 
constraints in completing assignments successfully is important in a highly technical 
culture. "Getting my work in on time" has a direct bearing on grades and gradua- 
tion. 

College/ School Involvement 

A strong correlation has been found between college involvement and success. 
College involvement can be characterized in several ways, such as living on campus 
or having part-time campus jobs (Astin, 1993), having friendship support, taking part 
in extracurricular activities, and interacting with college faculty. Campus involvement 
on the part of the student not only relates to academic success but also to identity 
achievement and to general satisfaction with the academic experience. 

Family Involvement 

Family encouragement of the first-year college student and/or participation of 
one's family in college appears to be an important factor in the student's success in 
college. Graduates of universities note that their families "just assumed" they would 
graduate. Early research suggested that persistent college students received more pa- 
rental advice, praise, and expressed interest (Trent & Ruyle, 1965). 



College Success Factors Index 



109 



Measuring Academic Aptitude 

The College Success Factors Index (CSFI) is an assessment instrument contain- 
ing 80 self-scoring statements designed to determine the readiness of high school 
juniors /seniors and college populations to complete their early years of college suc- 
cessfully. The following 10 statements are samples from the CSFI. Students are asked 
to agree or disagree with each statement on a five-point scale: "Strongly Agree" 
to "Strongly Disagree." They are also asked to respond as honestly as possible. 

• I can handle examination stress. 

• I involve myself with a lot of school or college projects. 

• People should stand up for what they believe. 

• Competition at college is necessary for success. 

• I know why my career choice requires a college degree. 

• I rarely feel that I am merely drifting along. 

• I always make time for my studies. 

• I like a class assignment that is challenging. 

• In group projects, I am often a leader. 

• My family will definitely attend my graduation. 

Scoring the CSFI 

The test can be self-scored or on a same-day basis offering individual print- 
outs within hours. The printout includes the student's total and sub-test scores and 
a bar chart to assist the student/ counselor/ instructor in understanding areas that are 
good, average, or need improvement. Scores below the statistical average fall below 
a so-called "watchline" or danger zone. Thus, students, counselors, and instructors 
have a visual indicator of strengths and weaknesses in the eight vectors of academic 
aptitude. 

Technical Background 

The technical data of the CSFI is very encouraging in terms of reliability and 
validity. The norm group has continued to expand over the years, and we now have 
20,000 students representing research universities, state universities, community col- 
leges, and private colleges. The mean scores for these norm groups have not varied 
more than .02% during the last eight years. Culture diversity studies were modeled 
after the diversity found in California college students. Checks over the last few years 
indicate initial footprints have been maintained in the growing norm group. 

Coefficient alpha and test-retest studies indicate coefficients of .91, which are at 
a very acceptable level of reliability. Validity studies, both concurrent and predictive, 
with a variety of students indicate coefficients of from -.30 to -.50, very desirable results 
for self-reporting instruments. (Negative signs indicate smaller scores and represent 
greater success). Coefficients were drawn between test scores and earned GPA. 

Using the CSFI 

Three versions of the CSFI are available: 

1. CSFI booklets: Students fill in their responses in paper booklet. The booklet 
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is self-scored, and the profile is provided in each booklet for the individual 
student and counselor /instructor. 

2. PC software for Windows: Large numbers of students can take the CSFI, using 
a reusable test booklet and a scan form. 

3. CSFI online: In association with Wadsworth Publishing, the CSFI is available 
on the Internet. Students can take the test using a computer at home or 
on campus. The test scores are printed out instantly with a bar chart and 
explanations for the eight factors. 

The Early Alert Retention Program allows colleges to identify high-risk students by 
name and CSFI watchline factor. Counselors and faculty interested in retention rates can 
use this program to assign FYE and intervention classes. 

Once a student's weaknesses have been identified by the CSFI, the instructor must 
be able to offer guidance concerning improvements. The eight vectors of academic apti- 
tude naturally suggest certain remedies: Students with poor responsibility scores need 
assertiveness training, just as those with poor scheduling skills need time-management 
training, and so forth. Instructors can offer their own interventions with the test, but 
the CSFI is designed to be used with a complete curriculum of improvements. The cur- 
riculum is based on a book. Making the Dean’s List (Hallberg, Achieris, & Hallberg, 2003), 
which includes 200 transparency-ready lessons and exercises. It is also available with a 
PowerPoint presentation for instructors. These presentations can be used as a 16-week 
total student development course or a single class module for instructors in a variety of 
subjects. 

Linking the CSFI to institutions includes programs in student development classes, 
college orientation, cohort classes, probation seminars, individual counseling, parent/ 
spousal orientation, classroom subject orientation. Educational Opportunity programs, 
high school and college Early Alert programs, and special population interventions such 
as nursing and research. 
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y of Student Engagement: 
of Effective Educational Practice 

A common question among those responsible for assessing 
first-year students and first year of college programs is "How 
do our students and programs compare to other institutions?" 
One way to answer this question is through benchmarking. 
Benchmarking is the process of continuously comparing and 
measuring an institution against high performing colleges and 
universities to help the institution take action to improve its 
own performance (Alstete, 1995). As it relates to the first year 
of college, benchmarking is adapting best practices to improve 
the quality of the first-year experience. 

Effective benchmarking does not happen by solely com- 
paring resources, reputation, or incoming student background 
characteristics. Although this kind of information reveals some 
useful facts about an institution, it does not tell us much about 
what is important to student learning — whether or not an insti- 
tution's programs and services are having the desired effect on 
students' activities, experiences, and outcomes. The National 
Survey of Student Engagement (NSSE) was designed with these 
purposes in mind. 



What is NSSE? (pronounced "nessie") 

Now in its third year, NSSE is a national effort to assess 
collegiate quality by collecting reliable, valid information about 
students' engagement in good educational practices directly from 
undergraduates at four-year colleges and universities. NSSE is 
supported by a grant from The Pew Charitable Trusts and co- 
sponsored by The Carnegie Foundation for the Advancement of 
Teaching and The Pew Forum on Undergraduate Learning. 

The NSSE instrument. The College Student Report, is ad- 
ministered each spring to random samples of first-year and senior 
students at public and private four-year colleges and universities. 
It can be completed via a traditional paper questionnaire or via 
the web. The random sampling method ensures that the results 
are comparable, meaningful, credible, and usable for institutional 
self-study and improvement efforts, as well as peer comparisons 
and national benchmarks. 

Unlike many other assessment surveys which require ex- 
pending a great deal of time, effort, and resources administering 
the survey and then figuring out what the results mean, 
NSSE is a "full service" survey and assessment provider. 
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NSSE handles the sampling and all aspects of data collection, including mailing 
surveys directly to students; collecting, checking, and scoring completed surveys; 
conducting multiple follow-ups with non-respondents; and generating custom- 
ized assessment and benchmarking reports that are distributed to participating 
colleges and universities. NSSE reports include a customized institutional profile 
that includes averages and response frequencies on all survey items, aggregated 
comparison data for both similar colleges and national norms, and a special insti- 
tutional comparison against NSSE's national benchmarks. 

National Benchmarks of Effective Educational Practice 

To help facilitate institutional benchmarking. The College Student Report fo- 
cuses on effective educational practices by assessing student engagement in five key 
areas. 

1. Level of academic challenge. Survey items in this benchmark include time 
spent preparing for class, amount of reading and writing, and institutional 
expectations for academic performance. 

2. Active and collaborative learning. Survey items in this benchmark include 
participating in class, working collaboratively with other students inside 
and outside of class, tutoring and so forth. 

3. Student-faculty interaction. Survey items in this benchmark include talking 
with faculty members and advisors, discussing ideas from classes with 
faculty members outside of class, getting prompt feedback on academic 
performance, and working with faculty members on research projects. 

4. Enriching educational experiences. Survey items in this benchmark include 
interacting with students with different racial or ethnic backgrounds or 
with different political opinions or values; using electronic technology; and 
participating in such activities as internships, community service, study 
abroad, co-curricular activities, or a culminating senior experience. 

5. Supportive campus environment. Survey items in this benchmark include 
the extent to which students perceive the campus helps them succeed 
academically and socially; assists them in coping with non-academic 
responsibilities; and promotes supportive relationships among students 
and their peers, faculty members, and administrative personnel and 
offices. 

These benchmarks capture key elements of college student engagement that 
research studies show to be important to student learning (Astin, 1993; Chickering 
& Gamson, 1987; Pascarella & Terenzini, 1991). National norms for these five bench- 
marks are based on responses of students from nearly 500 different institutions. It 
is important to note that NSSE provides the benchmark data to participating institu- 
tions and then it is up to institutional representatives to initiate the "benchmarking" 
process at their respective colleges and universities. 
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Using NSSE Data 

NSSE generates a number of user-friendly reports that can be immediately used and 
disseminated on campus to stimulate or focus conversations on the first year of college 
experience. Colleges and universities are using NSSE results in a number of ways: 

• Assessment and institutional improvement. As a diagnostic tool to identify 
areas in which the college or university can enhance effective educational 
practices and promote student learning 

• Benchmarking. As a frame of reference for comparing student performance 
among institutions with similar missions and academic programs, and for 
establishing national and sector norms of effective educational practice 

• Internal and external accountability. As a way of documenting changes in 
institutional effectiveness over time for accreditation, self-studies, and other 
purposes 

• Institutional research. As a source of information about students that can be 
directly linked with institutional records and results from other surveys to 
generate a more comprehensive picture of the undergraduate experience 

In addition, many campuses are putting NSSE data to use as it relates to the first-year 
experience. Examples include incorporating NSSE indicators into the evaluation of first- 
year learning communities, using NSSE data to assist with enrollment management and 
retention issues, incorporating NSSE into discussions related to academic advising, using 
NSSE data to assess general education requirements, and using NSSE to help establish 
student and faculty expectations related to student engagement in the first year of college. 
Additional examples of how NSSE is being used are included in Kuh (2001). 
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The Community College Survey of Student Engagement (CC- 
SSE) and The Community College Student Report, which reports 
its results, focus on institutional practices and student behaviors 
that demonstrate student engagement — and that research shows 
correlate highly with student learning and retention. In spring 
2003, 65,300 students responded to the survey, which is adminis- 
tered in randomly selected credit classes at CCSSE member col- 
leges. Details about the member colleges, student respondents, 
and results, as well as the survey sampling and administration 
process are made public at www.ccsse.org/ . 



CCSSE'S 2003 Benchmarks 



CCSSE has introduced a set of five benchmarks of effective 
educational practice in community colleges. The benchmarks 
provide significant opportunities for community colleges to 
gauge and monitor their performance in areas that are central 
to their mission and focus on teaching and learning. In addition, 
participating colleges can make appropriate and useful compari- 
sons between their performance and that of similar colleges. The 
benchmarks encompass 38 engagement items from the CCSSE 
survey, which address many of the more powerful contributors 
to effectiveness in teaching, learning, and student retention. 
They include: 

• Active and collaborative learning 

• Student effort 

• Academic challenge 

• Student-faculty interaction 

• Support for learners 

What are Benchmark Scores? 



Benchmark scores provide a useful way to look at an insti- 
tution's data by creating groups of conceptually related items 
that address key areas of student engagement. Using empirical 
results of factor analytic models and with the help of its Technical 
Advisory Panel, CCSSE grouped together survey items related to 
each of the five areas described above. In order to establish the 
benchmark scores, the items associated with a benchmark are first 
rescaled so that all items are on the same scale (0 to 1). Then the 
scores are standardized so that respondents' scores have a mean 
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of 50, weighted by full- and part-time attendance status, and a standard deviation of 25. 
Benchmark scores are then computed by averaging the scores on the associated items. 

Descriptions of CCSSE Benchmarks 

Active and Collaborative Learning 

Students learn more when they are actively involved in their education and have op- 
portunities to think about and apply what they are learning in different settings. The seven 
survey items that contribute to this benchmark include: 

During the current school year, how often have you: 

• Asked questions in class or contributed to class discussions 

• Made a class presentation 

• Worked with other students on projects during class 

• Worked with classmates outside of class to prepare class assignments 

• Tutored or taught other students (paid or voluntary) 

• Participated in a community-based project as a part of a regular course 

• Discussed ideas from your readings or classes with others outside of class 
(students, family members, co-workers, etc.) 

Student Effort 

Students' own behaviors contribute significantly to their learning and the likelihood 
that they will successfully attain their educational goals. Associated with this benchmark 
are eight survey items indicating student behavior in these terms: 

During the current school year, how often have you: 

• Prepared two or more drafts of a paper or assignment before turning it in 

• Worked on a paper or project that required integrating ideas or information 
from various sources 

• Come to class without completing readings or assignments (reverse coded) 

• Used peer or other tutoring 

• Used skill labs 

• Used a computer lab 

During the current school year: 

• How many books did you read on your own (not assigned) for personal en- 
joyment or academic enrichment? 

• How many hours did you spend in a typical week preparing for class (studying, 
reading, writing, rehearsing or other activities related to your program)? 

Academic Challenge 

Challenging intellectual and creative work is central to student learning and collegiate 
quality. Ten items from The Community College Student Report correspond to components 
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of academic challenge including (a) the nature and amount of assigned academic work, 
(b) the complexity of cognitive tasks presented to students, and (c) the standards faculty 
members use to evaluate student performance. These items are: 

During the current school year, how often have you: 

• Worked harder than you thought you could to meet an instructor's standards 
or expectations 

How much does your coursework at this college emphasize: 

• Analyzing the basic elements of an idea, experience, or theory 

• Synthesizing and organizing ideas, information, or experiences in new 
ways 

• Making judgments about the value or soundness of information, arguments, 
or methods 

• Applying theories or concepts to practical problems or in new situations 

• Using information you have read or heard to perform a new skill 

During the current school year: 

• How many assigned textbooks, manuals, books, or book-length packs of 
course readings did you read? 

• How many papers or reports of any length did you write? 

• To what extent have your examinations challenged you to do your best 
work? 

How much does this college emphasize: 

• Encouraging you to spend significant amounts of time studying 
Student-Faculty Interaction 

The more contact students have with their teachers, the more likely they are to learn 
effectively and to persist toward achievement of their educational goals. Personal interac- 
tion with faculty members strengthens students' connections to the college and helps them 
focus on their academic progress. The six items used in this benchmark are: 

During the current school year, how often have you: 

• Used e-mail to communicate with an instructor 

• Discussed grades or assignments with an instructor 

• Talked about career plans with an instructor or advisor 

• Discussed ideas from your readings or classes with instructors outside of 
class 

• Received prompt feedback (written or oral) from instructors on your 
performance 

• Worked with instructors on activities other than course work 
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Support for Learners 

Students perform better and are more satisfied at colleges that are committed to their 
success and cultivate positive working and social relationships among different groups 
on campus. Community college students also benefit from services targeted to assist them 
with academic and career planning, academic skill development, and other issues that may 
affect both learning and retention. The seven survey items contributing to this benchmark 
include: 

How much does this college emphasize: 

• Providing the support you need to help you succeed at this college 

• Encouraging contact among students from different economic, social, and racial 
or ethnic backgrounds 

• Helping you cope with your non-academic responsibilities (e.g., work, family) 

• Providing the support you need to thrive socially 

• Providing the financial support you need to afford your education 

During the current school year, how often have you: 

• Used academic advising/ planning services 

• Used career counseling services 

Using the Benchmarks 

There are a number of ways that college leaders might choose to use the benchmarks. 
In doing so, it is important to keep in mind that community colleges differ significantly — 
even dramatically — from one another on variables including size, location, resources, 
enrollment patterns, and student characteristics. It is important to take these differences 
into account when interpreting benchmark scores — especially when making institutional 
comparisons. 

These benchmarks are tools that community colleges can use to: 

• Convert data into useful information 

• Compare their performance to that of similar institutions, to the aspirations 
of their own faculty and staff, and to the full CCSSE population of community 
colleges 

• Compare their own performance across benchmarks 

• Identify areas in need of improvement and monitor the effects of improvement 
initiatives 

• Track their progress toward identified institutional goals 

Because the results are public, benchmarks also can stimulate conversation — within 
colleges and among policy makers — about effective education practice. 
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rst-Year Seminars 

The First- Year Initiative (FYI) benchmarking survey mea- 
sures learning outcomes from first-year seminars and guides 
improvement of these courses. The FYI instrument and assess- 
ment procedures were developed through a partnership between 
the Policy Center on the First Year of College and Educational 
Benchmarking, Inc. with funding from The Pew Charitable Trusts 
and The Atlantic Philanthropies. The launch of FYI in 2000 is a 
signal of the changing focus in developing first-year seminars. 
During the last decades of the 20th century, much of the focus of 
higher education was on establishing first-year seminars at col- 
leges and universities. National surveys show that most colleges 
and universities now have established some form of first-year 
seminar. The challenge for the new century is to improve the 
effectiveness of these courses to ensure the greatest achievement 
of course goals. Continuous evaluation of course outcomes and 
evolution of course content and pedagogy will likely be needed 
to adjust seminars to the changing needs of new students. 

FYI is administered to first-year students in the last week 
of their first-year seminar. The survey measures 15 constructs — 
built as statistically validated factors. Ten of the measures evalu- 
ate learning outcomes from first-year seminars, especially about 
the impact on other courses in which students are also enrolled. 
In addition, the instrument measures three constructs about the 
delivery of the seminar and two additional measures about the 
student's overall satisfaction with the institution. 



Course Learning Outcomes 

• Study strategies 

• Academic /cognitive skills 

• Critical thinking 

• Connections with faculty 

• Connections with peers 

• Out-of-class engagement 

• Knowledge of campus policies 

• Knowledge of wellness/ spirituality 

• Managing time /priorities 

• Knowledge of wellness 

Assessment of Seminar Delivery and Effectiveness 

• Usefulness of course readings 
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• Presence of engaging pedagogy 

• Overall course effectiveness 

Satisfaction with Institution 

• Satisfaction with college / university 

• Sense of belonging / acceptance 

Unlike most end-of-course, locally-produced "teacher/ course evaluations," The FYI 
survey instrument was developed by experts in first-year seminars and pilot tested with 
more than 2,000 students. The centerpiece of an FYI study is not the instrument itself. In 
other words, the instrument is not designed to judge the performance of a single seminar 
section or instructor in isolation. Rather, FYI is designed to offer a context for assessment data 
by providing comparisons with outcomes from similarly situated courses in two ways. 

First, institution-level data are compared with data from six selected "peer" institu- 
tions. These institutions closely match in terms of institutional characteristics and course 
formats. The use of peer institutions ensures that comparisons are "apples to apples." 
For example, the outcomes from a one contact-hour course would not be expected to be 
the same as the outcomes from a three contact-hour course, so the only useful comparison 
would be between seminars using the same number of contact hours. 

Second, data are also analyzed within each participating institution so that the out- 
comes from each section are compared with outcomes from all other sections. In this case, 
the benchmarks are between sections rather than between institutions. This analysis allows 
course administrators to identify high-performing sections and to learn from the organiza- 
tion and delivery structures used in those sections. 

A psychometrically sound instrument and appropriate data context are necessary but 
not sufficient components of a successful course assessment strategy. In addition to these 
elements, the survey process must identify which course factors have the greatest impact on 
overall effectiveness on the first-year seminar. The FYI benchmarking study can produce a 
decision matrix created through a sophisticated data analysis model to identify the highest 
impact factors. The decision matrix is a 2 x 2 table which plots the level of achievement of 
each factor and its level of importance in overall course effectiveness (Figure 1). 

The decision matrix conforms to the qualities of good assessment noted earlier in this 
monograph in that it provides confirmation of existing practices which are working well 
and identifies areas where improvement would have the greatest positive impact. These 
data and the decision matrix are particularly useful in increasing the impact of first-year 
seminars on average — the first concern of seminar directors. But if seminars are to fulfill 
their promise of serving all students, it is equally important to be able to disaggregate 
seminar evaluations by a variety of student characteristics. 

Because FYI serves as a course evaluation the instrument does not collect student 
names or identifiers to ensure that students feel completely free to state their honest opin- 
ions. Without individual student identifiers the FYI data cannot be linked to other exist- 
ing data sources, so demographic characteristics of interest must be collected on the FYI 
survey instrument. The demographic data collected by FYI may be used to disaggregate 
course results so as to establish the course's impact on subgroups of students within each 
demographic stratum. Demographic data include: 

• Gender 

• SAT / ACT score 
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• Ethnic group 

• Age 

• High school grade point average 

• First-term grades in other courses 

• Source of financial aid 

• Time spent studying/ working/ participating in campus events per week. 

• Living arrangements (on/ off campus) 

• Use of alcohol 



Low Factor Scores 


High Factor Scores 


High Level of Impact on Course 
Effectiveness 


High Level of Impact on Course 
Effectiveness 


(Improvement in these factors will have 
the greatest positive impact on the course.) 


(These factors are having a positive impact 
on the course and should be maintained.) 


Low Factor Scores 


High Factor Scores 


Low Level of Impact on Course 
Effectiveness 


Low Level of Impact on Course 
Effectiveness 


(Improvement in these factors will have 
little positive impact on the course.) 


(There is little room for improving these 
factors, and improvement would have 
little positive impact on the course.) 



Figure 1. Decision Matrix for FYI 

Disaggregating FYI data by any specific demographic variable can be helpful — for 
example determining if men and women are having similar experiences in first-year semi- 
nars would be of interest to most seminar administrators. Demographics analyses are 
of greater importance when several variables are combined to disaggregate the findings 
further. For example national data suggest that even among female students, the seminar 
may be differently perceived by specific ethnic (e.g., Hispanic, White, African American), 
age, and ability groups (i.e., based on high school grades or SAT / ACT scores). 

Conclusion 

FYI has a number of advantages over most locally produced, end-of course evaluations. 
The external validation of the instrument gives it credibility and increases the possibility 
that findings will be taken seriously. The advantages go beyond the survey instrument 
itself. An FYI survey is particularly capable of producing change because the data are 
presented in an appropriate context — either through comparison with findings from peer 
institutions or via comparisons across sections of the course within a single institution. 
Because participating institutions receive the raw data from the FYI study, it is possible to 
focus analyses on a large range of student subpopulations which are of local interest. 



Looking at High-Risk Behaviors 

Increased alcohol use and the accompanying negative con- 
sequences of such use have powerful impacts on the first-year 
experience. Even if a student is not one of those who, according 
to the most recent study from the Harvard School of Public Health 
(Wechsler, Lee, Kuo, Seibring, Nelson, & Lee, 2002), is missing 
class as a result of drinking the night before (30%) or blacking out 
(27%), he or she is probably one of those whose sleep or study- 
ing is being interrupted by drinkers (60%) or who is having to 
baby-sit friends who have had too much to drink (48%). 

The above figures from the Harvard study are from a na- 
tionally representative dataset and usually show great variability 
from campus to campus. National statistics are helpful in deter- 
mining how your campus compares to others; but because each 
campus climate is unique, one cannot assume that a national 
problem is also a local problem. With limited resources for educa- 
tion and prevention, it makes sense to target those resources on 
specific issues that are demonstrable problems. For example, a 
well-designed survey might reveal that key issues for a particular 
campus are impaired driving or inadequate training of bartenders 
on recognizing signs of intoxication. Thus, resources for alcohol 
and drug education might be directed toward these issues. 

There are many valid reasons to survey first-year students 
about high-risk behaviors. Surveying students as they enter col- 
lege helps institutions better separate pre-college experiences 
from college experiences and to program accordingly. Surveying 
your first-year students as part of an ongoing assessment of all 
classes (e.g., a representative sample of all students in the spring 
term) will help illuminate how substance use and other high-risk 
behaviors change as students pass through college. 

A number of instruments are available for examining pat- 
terns of alcohol / drug use and other high-risk behaviors among 
college students. Institutions also have the option of creating their 
own. A locally developed instrument ensures that particular 
institutional concerns are addressed and is more likely to reflect 
the unique institutional culture, but an effective instrument 
can be very difficult to develop and should not be tackled by a 
novice. A good reason to use the available instruments is that 
they allow institutions to compare their students' responses with 
national norms, and this might not be possible with a locally 
grown survey. 

The Core Alcohol and Drug Survey was very popular in the 
1980s and 1990s, mostly because of its roots in and promotion by 
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the U.S. Department of Education's Fund for the Improvement of Post Secondary Educa- 
tion (FIPSE) alcohol branch. It is an inexpensive and low-tech effort to examine alcohol 
and other drugs and their effects. The Core is available for paper and pencil administration 
only. Participating institutions order copies of the Core survey from the Core Institute and 
administer the survey themselves and mail in the completed instruments. Several months 
later the institution receives a frequency report of the responses and can also request an 
executive summary of the responses, cross-tabulations on several key demographic variables 
(e.g., sex, race, age), and a computer file containing the unit-record responses. 

More schools are recognizing the need for broader assessment now and are using 
instruments that include sexual behavior, eating behavior, violence, and other areas of risk 
for college students beyond just alcohol and other drugs. The American College Health 
Association developed the National College Health Assessment (www.acha.org/ projects_ 
programs/ assessment. cfm) as a comprehensive health survey for college students. The 
survey includes alcohol, tobacco, and other drugs, but also covers sexual health, mental 
health, injury prevention, nutrition, and other areas. It is available in both paper-and- 
pencil and web versions. The paper-and-pencil forms are machine scannable and are 
mailed to a central source to be scanned. Several months later participating institutions 
will receive a frequency list and an executive summary and can request a data file and a 
reference group report. 

TheHealthSurvey is another comprehensive survey for college students, examining 
alcohol, tobacco, and other drugs as well as sexual behavior, eating behavior, violence and 
safety, suicide, and use of campus education, prevention, and counseling sources, among 
other topics. While there is overlap with the topics covered in the NCHA, the emphasis in 
TheHealthSurvey is less on issues of general health (such as injury prevention and what 
vaccinations the student has received, as in the NCHA) and more on areas of college student 
risk. A unique feature of TheHealthSurvey is an optional "Social Marketing" evaluation 
component. This feature allows schools to provide graphic files of social marketing materi- 
als (such as a "social norms" poster) that will be embedded in the web-based questionnaire 
in order to obtain feedback from students. TheHealthSurvey was designed exclusively for 
Internet-based administration and is not available in a paper version. Institutions provide 
student e-mail addresses to the company that created the survey, which administers the 
survey to students. E-mail addresses are kept confidential and are erased after the survey 
project is completed. The turn around time for TheHealthSurvey is shorter than the previous 
two surveys examined, often about a month from the end of the survey until the reporting 
is received. Participating institutions receive a frequency list and the unit-record data in 
SPSS format. They can also choose from a menu of other reporting options, including a 
PowerPoint presentation of their data, an extensive list of crosstab options for sub-group 
reporting, an executive summary, and customized analyses. 

One of the more well-known alcohol surveys is the Harvard School of Public Health 
survey. This nationally representative study is frequently quoted in the media, and 130 
campuses across the country have participated in this research project. Until recently, 
the survey was available only to the institutions involved in the study. Now, research- 
ers at the Harvard School of Public Health have created a comprehensive health survey, 
based on the College Alcohol Study, called the Study of College Health Behaviors (http: 
/ / www.hsph.harvard.edu/cas/ chb/) and scheduled to be released in spring 2004. Pro- 
motional materials suggest that the instrument will closely resemble the NCHA topics. 

The cost to participate in these survey projects varies considerably. Because of the 
many options available, such as paper versus web administration, how many follow up 
contacts one initiates, and the various reports, it is difficult to put a specific number on 
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each survey package. In general terms, however, the Core survey is the least expensive 
option, the NCHA and TheHealthSurvey are in the moderate category, and the Study of 
College Health Behaviors is in the expensive category. Specific information is available at 
each project web site. 

In summary, surveying first-year students helps campuses identify the presence of 
high-risk health behaviors associated with attrition and academic failure. Various instru- 
ments are available to help campuses explore these issues, with the field moving toward a 
more comprehensive health survey that is administered over the Internet. Each instrument 
has its own strengths and should be assessed individually by each campus with respect to 
the most appropriate questions for particular campus culture and concerns, budget, and 
format. 
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Successful institutions share three basic attributes: They 
focus on the needs of their students, they continually improve 
the quality of the educational experience, and they use student 
satisfaction data to shape their future directions. 

Student satisfaction studies measure how effectively 
campuses deliver what students expect, need, and want. 
These self-examinations enable institutions to measure their 
students' satisfaction with a wide range of college experiences. 
By taking "soundings" of student satisfaction, institutions are 
able to pinpoint their strengths, as well as areas in need of 
improvement. 

Traditionally, colleges and universities have measured 
one dimension of student satisfaction only — institutional 
performance. However, for greatest impact and accuracy, 
satisfaction should be viewed within the context of student 
expectations or level of importance. For example, the quality 
of food service and the use of student activity fees repeatedly 
surface as areas of high dissatisfaction for students. But when 
asked to indicate the importance of these areas to their over- 
all educational experience, students rate both items relatively 
low. Likewise, student parking has also been an area of high 
dissatisfaction, but the level of importance varies by type of 
institution. Students at primarily residential campuses rate 
parking with lower importance than students at institutions 
with a large commuter population. Thus, the interrelationship 
between expectations and levels of satisfaction is crucial to a 
better understanding of student behavior. 

Principles of consumer theory serve as the basis for this 
approach to student satisfaction assessment. Students are 
viewed as consumers who have a choice about whether to in- 
vest in education and where to enroll. In addition, students are 
seen as individuals who have definite expectations about what 
they want from the campus experience. From this perspective, 
satisfaction with college occurs when the expectation is met or 
exceeded by an institution. If strongly held expectations are 
not met, it is likely that a student will leave an institution in 
search of one that can meet the expectations. Consequently, 
it is critical for institutions to focus on both student expecta- 
tions, as well as levels of satisfaction, in order to ensure that 
their students are satisfied and, ultimately, that students are 
retained. 
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How Does This Approach to Student Satisfaction 
Assessment Work? 

With assessment tools that embody this approach, such as the Student Satisfaction 
Inventory™, students rate each item by the importance of the specific expectation as well 
as their satisfaction with how well that expectation is being met. Students are asked to 
respond to two questions instead of one. The usual question, "How satisfied are you?" is 
accompanied by an additional question, "How important is it to you?" This second ques- 
tion makes all the difference because it addresses the heart of students' concerns and allows 
students to indicate what really matters to them. 

Three scores are calculated for each item: (a) an importance score, (b) a satisfaction 
score, and (c) a performance gap score. Importance score ratings reflect how strongly stu- 
dents feel about the expectation (the higher the score, the more important it is to a student, 
hence the stronger the expectation). Satisfaction ratings show how satisfied students are 
that the institution has met the expectation (the higher the score, the more satisfied the 
student). 

The power of this approach lies in the third score — the performance gap. Individu- 
ally the importance scores and the satisfaction scores are quite helpful, but taken together, 
they provide invaluable information about how well the institution is meeting students' 
expectations. The performance gap score is the difference between the importance rating 
and the satisfaction rating for each item. For example, if a student gave an importance rat- 
ing of 6 and a satisfaction rating of 2, the performance gap for that item would be "4." A 
negative performance gap results when students rate satisfaction higher than importance. 
This typically happens only with areas that are not highly valued by students. 

Maximizing the Use of All Measures 

With this approach to student satisfaction assessment, it is essential to use all the informa- 
tion provided. It is especially vital that all three areas of measurement (importance, satisfac- 
tion, and performance gaps) be considered when listening to students' voices. Focusing on 
only one piece of information (for example, performance gaps) will not necessarily ensure 
that you are concentrating on the items that students consider most important. By looking 
first at how important an item is to students and then looking at the largest performance 
gaps in those areas of greatest importance, you have a priority-ranked list of leading issues 
that could dramatically improve students' satisfaction if addressed. 

The items on the survey are grouped statistically and conceptually into scales that repre- 
sent various functions such as advising, instruction, registration, and financial aid. The scales 
provide a quick overview of areas that need improvement, while individual items pinpoint 
areas of concern within a particular area or campus function. The campus can use the scale 
information to identify the broad perspective in relation to other scale areas and then look 
closer at individual items within the scale to determine specific issues that need attention 
within that area. For example, the Campus Services scale represents an array of campus ef- 
forts that support students in their academic endeavors. This scale covers everything from 
adequacy of the library and computer labs to the availability of tutoring. As shown in the 
example in Table 1, the scores for all items do not carry equal weight within the scale. In fact, 
the two items that are most important to students, and for which there are the highest expec- 
tations, also have the largest performance gaps — adequacy and accessibility of the computer 
labs, followed by the adequacy of career services. These data reveal the trouble spots in the 
campus services area where student expectations are not being met. 
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Table 1. 

Scale Example for Campus Services 



Scale /Item 


Importance 


Satisfaction 


Perfomance 

Gap 


Campus Services 


5.98 


5.10 


0.88 


Computer labs are adequate and accessible. 


6.29 


5.12 


1.17 


There are adequate services to help me 
decide upon a career. 


6.16 


4.91 


1.25 


Library resources and services are adequate. 


6.10 


5.03 


1.07 


Academic support services adequately meet 
the needs of students. 


5.98 


4.95 


1.03 


Tutoring services are readily available. 


5.88 


5.17 


0.71 


Bookstore staff are helpful. 


5.78 


5.21 


0.57 


Library staff are helpful and approachable. 


5.70 


5.32 


0.38 



Rating Scale: 1 = low score 7 = high score 

What Do the Results Mean? 



The matrix in Figure 1 provides a graphic conceptualization of how student expecta- 
tions (importance) and satisfaction are considered together when pinpointing institutional 
strengths and opportunities for improvement. 



Very 

Important 



Very 

Dissatisfied 



Very 

Satisfied 



Very 

Unimportant 



♦ High importance /low satisfaction - pinpoints areas that should 
claim the institution's immediate attention 

• High importance /high satisfaction - showcases the institution's areas 
of strength that should be highlighted in promotional materials 

♦> Low importance / low satisfaction - presents an opportunity for the 
institution to examine those areas that have low status with students 

■ Low importance /high satisfaction - suggests areas from which it might 
be beneficial to redirect institutional resource commitments to areas of higher 
importance 



Figure 1. Matrix for Prioritizing Action 
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Taking Student Satisfaction to the Next Level 

Campuses can enhance their understanding of the institutional situation by expanding 
their assessment to include all campus constituents. In addition to students, this includes 
faculty, staff, administrators, and board members. Using an instrument such as the Institu- 
tional Priorities Survey™, institutions can pinpoint even more precisely those areas where 
improvements in campus programs and services can impact the level of student satisfaction. 
This expanded assessment also serves as a campus climate assessment to see where students 
and campus personnel value the same areas and where perceptions differ. By quantifying 
the importance faculty, staff, and administrators place on student expectations, as well as 
their perception of student satisfaction, campus leaders are able to pinpoint more clearly 
the priorities for action. It also allows the institution to identify opportunities for further 
dialogue on what truly matters to students and what campus personnel value. 

The Institutional Priorities Survey asks campus personnel to indicate how important 
they believe it is that the institution meets student expectations and their level of agree- 
ment that the institution is meeting the expectation. Campus personnel respond with an 
agreement score rather than a satisfaction score because they are indicating a perception 
they have about the student experience, rather than a satisfaction level they have from 
their own experience. 

Figure 2 illustrates the identification of common strengths and priorities using the 
combination of student importance/ satisfaction results and campus personnel importance/ 
agreement scores. 




The areas of greatest institutional strength 

A. Items of highest importance /highest satisfaction (student satisfaction 
data) 

B. Items of highest importance /highest agreement (campus personnel data) 

C. Intersect of A & B = areas of greatest strength 

The areas of highest institutional priority 

A. Items of highest importance /lowest satisfaction (student satisfaction 
data) 

B. Items of highest importance /lowest agreement (campus personnel data) 

C. Interest of A & B = areas of highest priority 



Figure 2. Identifying Common Strengths and Priorities 
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It is important to review both the areas where overlap occurs and the areas that are 
viewed uniquely by each group on campus to better understand the overall campus climate. 
The areas viewed uniquely provide the opportunity for further discussion on campus. 

How Do Campuses Use the Results? 

Because this approach to student satisfaction generates a significant amount of useful 
information, institutions find themselves in the unique position to determine: 

• The aspects of the college or university that are of greatest importance to their 
students and contribute most to their satisfaction 

• The areas where they are already meeting or exceeding student expectations 

• The areas of greatest institutional strength 

• Aspects of the institution with the most severe performance gaps 

• Areas that are perceived differently by students and campus personnel 
when combined with the broader institutional assessment 

• Modifications or adjustments needed to create a more student-centered 
campus 



Why Do Campuses Embrace This Approach? 

The assessment of student expectations and satisfaction provides institutions with the 
information they need to be responsive to student perceptions. Students whose needs are 
actively addressed by their institution are more likely to be successful in achieving their 
educational goals and more likely to persist. Campuses must continue to follow the lead 
of their business counterparts and take "soundings" of student satisfaction if they are to 
serve their students and remain competitive. Student satisfaction data can be the driving 
force for informed decision making on the campus. Many campuses have found that the 
combination of satisfaction and importance data is a key element for strategic planning, 
enabling them to identify initiatives yielding the greatest impact on student success. 
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Cognitive Tests 
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Marc Cutright The assessment of critical thinking by college students is an 

intriguing, enticing goal. After all, it is arguably the core purpose of 
what we do; it is what makes our education "higher." Demonstra- 
tion of the development of critical thinking would address concerns 
that students are passive recipients of information. The challenges 
of democratic citizenship in the information age will tax critical 
thinking capabilities. High-order thinking skills are of increasing 
interest to accrediting bodies and state legislatures, among other 
external constituents. These issues are particularly acute for first- 
year-student educators; the skills and attributes of critical thinking 
are explicit goals for many first-year seminars and core curricula. 

However, the challenges of assessing critical thinking are 
substantial and begin with our very definitions of the concept — or 
even our failures to define it. In the mid-1990s, a large sample 
of California faculty members affirmed the importance of critical 
thinking as an educational outcome. Eighty-nine percent of the fac- 
ulty said critical thinking was a "primary objective" of their teach- 
ing — but only 19% could give a clear explanation of the concept. 
While 78% of these faculty said that students "lacked standards" 
to assess their own thinking, only 8% could name or describe any 
of these standards (Paul, Elder, & Bartell, n.d.). At about the same 
time on the East Coast, 88% of institutions in South Carolina (N = 
60) indicated that the development of critical thinking was a key 
institutional goal — but only 46% had defined critical thinking (Cook 
et al., 1996). 

While it is probably best that the assessment of critical thinking 
begin with an institutional discussion of its particular definition in 
that context, some well-accepted definitions and theories lend guid- 
ance. For example. Bloom's Taxonomy (1956) ranges, on the low 
end, from memorization, listing, definition, and so on, to the ability 
to evaluate, assess, and judge at more sophisticated levels. The 1990 
Delphi Report on critical thinking, endorsed by an expert panel from 
a variety of disciplines, defined critical thinking as "purposeful, 
self -regulatory judgment which results in interpretation, analysis, 
evaluation, and inference, as well as explanation of the evidential, 
conceptual, methodological, criteriological, or contextual consider- 
ations upon which that judgment in based. . ." (Facione, 1990, p. 3). 
Robert Ennis (n.d.), developer of the Cornell Critical Thinking Test, 
says critical thinking "means reasonable and reflective thinking on 
deciding what to believe or do," and he holds that it involves both 
dispositions and abilities. Other definitions abound. 
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Beyond definitions, other questions have to be answered before critical thinking can 
be meaningfully assessed on a campus. Among them: 

• Is the campus interested in dispositions to critical thinking or skill 
attainment? 

• Is the campus interested in the assessment of critical thinking as a pre-test, 
post-test, pre-post combination, instructional goal, or some combination 
thereof? 

• Will key stakeholders accept that no single assessment will tell them 
everything about individuals' thinking skills or is fully dependable? 

• Have those charged with assessment considered circumstances that may 
affect skill attainment other than classroom instruction? Maturation 
itself increases critical thinking to some degree, regardless of formal 
education. 

• Have assessment professionals on the campus considered the confounding 
of critical thinking with writing ability, math skills, vocabulary, or other 
discreet concepts? A student's ability to construct a conventional essay, 
for example, may be a matter of mastering a formula for such rather 
than putting real critical thinking into it, and the two are often confused. 

How important is it to you to isolate critical thinking from these other 
concepts and abilities? Conversely, do key stakeholders on the campus 
hold the position that writing ability is integral to critical thinking and its 
demonstration? 

• Does the campus support the chosen approach, including alignment of 
definitions and assessments with the curriculum? 

• Is quantitative reasoning part of the campus' critical thinking concept 
and goals? Some definitions and instruments to assess critical thinking 
include quantitative reasoning skills, while other definitions and 
assessments consider them distinct issues. 

• Is there sufficient faculty support for critical thinking as a goal, and is 
there sufficient involvement in the assessment of critical thinking, that 
changes in programs, instruction, and curriculum might come about from 
assessments? As is generally true in student assessment, measurement 
without feedback to instruction is unlikely to lead to any improvement. 

• Will the assessment environment for students call forth their best, honest 
efforts to do well? For example, a high-stakes pre-test environment, 
perhaps for the purpose of section placement, combined with a low- 
stakes post-test environment, would likely yield relatively insignificant 
evidence gains. 

Once these and other key questions can be resolved to the satisfaction of primary 
campus constituents, the selection of instruments is key. Vendors for each of the instru- 
ments noted below — and the list is not exhaustive — can provide substantial information 
on reliability, validity, and the definition of critical thinking or a closely related concept 
on which the instrument is based. Costs are difficult to cite with meaning; charges vary 
not only on the length of the instrument, but on such factors as whether the instrument 
is location-scored or returned for such to the vendor, the complexity of the testing report, 
the numbers of individuals tested, and whether essays are involved. A very rough range 
for testing costs would be from $4 per student to more than $20. 
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Instruments Assessing Critical Thinking 

• The Watson-Glaser Critical Thinking Appraisal, from The Psychological 
Corporation, comes in two basic formats, a 40-question form taking 45 
minutes, and an 80-question form taking 60 minutes. Both forms use 
multiple-choice questions and passages of text for evaluation. The Watson- 
Glaser produces a single score based on the abilities of inference, recognition 
of assumptions, deduction, interpretation, and evaluation of arguments. 

• The Cornell Critical Thinking Test, from Critical Thinking Books & Software, 
has a generally targeted test for "grade 5 to adult," developed by Robert 
Ennis. The Level Z test, the variation probably most appropriate for college- 
level testing, looks at induction, deduction, credibility, identification 
of assumptions, semantics, definitions, and prediction in planning 
environments. 

• ACT's Collegiate Assessment of Academic Skills (CAAP) is targeted to "post- 
core" students in higher education (as are The Academic Profile and College 
BASE, summarized below). The CAAP measures writing (through an essay), 
reading, math, science reasoning, and critical-thinking. The critical-thinking 
test measures the ability to clarify, analyze, evaluate, and extend arguments. 

It is multiple choice, tied to text passages. The CAAP is available in 
individual modules. 

• The Academic Profile of ETS comes in a standard, two-hour, 108-question form, 
on which both group and individual scores are reported; the abbreviated 
40-minute, 36-question form yields group data only. Area tests in the 
humanities, social sciences, and natural sciences each include measurements 
of reading and critical-thinking ability. There is an essay option. 

• The University of Missouri's College BASE has test content on English, essay 
writing, math, science, social studies, and reasoning competencies. The 
latter is assessed in ascending levels of reasoning: interpretive, strategic, 
and adaptive. The reasoning competency questions are embedded within 
the domain tests. Modules can be selected. A more detailed description of 
College BASE is available elsewhere in this volume. 

• Insight Assessment, formerly California Academic Press, offers two tests of 
particular interest. Both are based on the Delphi definition of critical-thinking. 

The California Critical Thinking Skills Test is targeted to undergraduates, 
graduate students, and professionals. Several forms are available, all of 
them include 34 multiple-choice items, take 45 minutes to complete, and 
are discipline-neutral in content. The California Critical Thinking Disposition 
Inventory — disposition being distinct from skills — has scales on truth 
seeking, open-mindedness, analyticity, systematicity, critical-thinking self- 
confidence, inquisitiveness, and cognitive maturity. The instrument is 75 
items and is Likert-scaled, taking about 20 minutes. 

Several institutions and college systems have made notable, independent efforts to 
assess and drive critical thinking, and their work is available for examination and emula- 
tion. 



The University of South Florida's Cognitive Level and Quality Writing 
Assessment Instrument (CLAQWA) is based on Bloom's Taxonomy. Writing 
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and the demonstration of cognitive skills are assessed independently based 
on the same writing sample. A more detailed description of CLAQWA is 
available elsewhere in this volume. 

• The Minnesota Community Colleges have developed the Critical Thinking 
Inventory, based on Robert Ennis's work and on adaptive interviewing 
techniques from Cathy Baumann. The interview approach is based on models 
of the assessment of second language oral proficiency. The inventory can be 
targeted to subject areas or can be interdisciplinary, and it assesses a variety 
of types of reasoning, from inductive generalization to values reasoning. 

• The Critical Thinking Project and Rubric at Washington State University has 
been funded by the state, the Fund for the Improvement of Post Secondary 
Education, and various foundations. The rubric yields a score for a writing 
sample and focuses on problem /issue identification, recognition of one's 
own perspective, recognition of the perspectives of others, assumptions, 
supporting evidence, context, and conclusions. Its proponents hold that 
its main benefit has been as an impetus to the adoption of particular 
instructional strategies, by making the particular elements of critical thinking 
more explicit for classroom and assignment incorporation. 

Whatever means of assessment are used, the topic of critical thinking will continue to 
grow in centrality to our assessment efforts, and interest in the development of new means 
of assessment will remain keen. 
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College BASE-lining Your First-Year Students 



Pamela A. 
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Recent research by the Policy Center on the First Year of Col- 
lege at Brevard College emphasizes the importance of a student's 
early collegiate experience in succeeding during subsequent 
post-secondary years. Surveys and questionnaires evaluate many 
important components of student life, while testing instruments 
assess progress in academic achievement. The development of basic 
skills provided in the general education curriculum for beginning 
students supports their completion of academic goals in their ma- 
jor field of study and provides the groundwork for post-graduate 
study and attainment of career goals. 

Assessment activity is often initiated on campuses to improve 
retention and graduation rates. While retention and graduation 
rates are extremely important goals, the rates alone do not indicate 
individual academic progress and success in general education, 
nor do the rates reflect institutional effectiveness. The College 
Basic Academic Subjects Examination (College BASE or CBASE) 
is one instrument designed for assessing knowledge and skills in 
general education. While a standardized test is only one tool for 
assessment, scores from standardized tests, when triangulated 
with other quantitative and qualitative assessments, can be useful 
in evaluating the first year in college and in providing a baseline 
for later assessments. 

College BASE was developed at the University of Missouri- 
Columbia in the mid-1980s to 



• Respond to concerns about the quality of post- 
secondary education 

• Contribute an assessment instrument in an area 
where few existed 

• Respond to concerns that college entrants may not 
have the ability to handle college-level work 

• Provide a statewide pre-admissions requirement for 
students advancing to teacher education programs 

College BASE was first used in 1988 as a teacher education 
requirement in Missouri. Because of the broad skill areas tested 
and the universality of those skills to other fields of study. Col- 
lege BASE is also used to assess campus-wide general education 
achievement. Currently, the exam is used by approximately 150 
institutions nationwide. The users include both community colleges 
and four-year institutions whose Carnegie designations range from 
baccalaureate college to research university. 
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College BASE is a criterion-referenced achievement test that assesses a student's knowl- 
edge and skills in English, mathematics, science, and social studies. The exam is made up 
of 180 multiple-choice items and includes an optional essay prompt. Each subject area tests 
skills and sub-skills within clusters. For example, the English subject area tests skills in the 
two cluster areas of Reading and Literature and Writing. The Reading and Literature cluster 
contains three skill areas: (a) reading accurately and critically, (b) reading analytically, and 
(c) understanding a range of literature. Most items in this cluster require the examinee to 
read a given passage and then answer questions about what has been read. 

Similarly, in mathematics there are three cluster areas: (a) general math, (b) algebra, and 
(c) geometry, each with 2 to 3 identified skills. A complete guide to test content appears on 
the College BASE web site at www.arc.missouri.edu/ collegebase/ The guide to test content 
provides a description of all subjects, clusters, skills, and sub-skills, with sample items. 

Each student and the institution receives a copy of the student's score report. The score 
report provides scaled scores for each of the four subjects and the 11 clusters. Designations of 
"high," "medium," and "low" are provided for each skill. A composite score is also provided 
when a student takes all four subjects. The institution also receives a report summarizing 
the performance of its students in the subjects and clusters described above. 

Administration of the four subject areas to all students requires approximately three 
hours, with an additional 40 minutes when the essay assignment is used. When assess- 
ment time is limited, institutions may administer the College BASE 1:1 (short form). This 
alternative administration method is useful when composite institutional results are more 
important than individual results. Each student is assigned one or two multiple-choice 
subjects, spiraled to provide four-subject results. Students receive score reports for the 
subject(s) taken, and the institution receives a summary of scores for all four subjects, as 
well as copies of all student score reports. 

One of the most important uses of College BASE is to provide baseline data for colleges 
and their students. Giving the test to incoming first-year students allows the institution 
to identify individual students' relative strengths and weaknesses, which is a necessary 
first step in planning for remediation and improving students' chances for success. An 
additional administration of College BASE, when students have completed their general 
education requirements, will reveal value-added improvements in achievement of the 
knowledge and skills measured by College BASE. 

When College BASE scores are compared with classroom grades and other qualitative 
assessments, an analysis of a student's progress and achievements is more complete. With 
performance information from several different sources, including scores from a standard- 
ized test, students are afforded additional opportunities for self-evaluation and reflection, 
and advisors are provided with an additional data point for counseling. If College BASE 
is administered a second time in the student's undergraduate experience, a comparison 
of scores from both administrations should show growth in the areas covered by College 
BASE. 

At the institutional level, advisors and faculty receive important data when the exam 
is administered during the first year of college. Composite data from first-year students 
provide information about the preparedness of the students for college work, especially 
important when students in a new class arrive with varying degrees of secondary school 
preparation. 

By examining institutional composite scores longitudinally, institutional researchers 
and faculty have base data to complete curriculum review and to plan improvements. Lon- 
gitudinal data also provide accountability evidence often required by regional accreditation 
bodies, state legislatures, taxpayers, or tuition-payers. 
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In addition, an institution may compare its results with composite scores from a peer 
group of institutions or with scores from a national normative group. While a single as- 
sessment instrument will not provide all of the information an institution needs to evalu- 
ate its effectiveness, gathering baseline information on first-year students will provide a 
foundation for further study. 

College BASE may be especially useful to institutions that are concentrating more ef- 
forts on the academic and personal development of first-year students. By collecting data 
and providing results, colleges can help ensure that students are better-informed and better- 
motivated, provide faculty feedback for curriculum review, and offer administrators data 
on student performance to integrate with other measures of institutional effectiveness. 
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A major challenge confronting higher education in the 
21 st century is assessing student learning in general educa- 



tion. This issue begins with the question of what comprises 
general education? Faculty have been debating this issue for 
years, and the purpose of this essay is not to continue that 
debate but to assume that general education consists of knowl- 
edge and skills that provide the foundation upon which all 
other learning follows. Accordingly, general education is that 
which includes such topics as communication (reading, writ- 
ing, speaking, listening), calculation (mathematics through 
college algebra), reasoning and critical thinking, computer 
skills, cultural awareness, and the like. Once defined, the next 
questions are how do we know that students have mastered 
general education and has our institution contributed to that 
mastery? 

These questions have led to requests by postsecondary 
institutions for testing companies to develop tools that can be 
used to assess a number of components of general education 
knowledge. ACT, Inc., a nonprofit educational testing orga- 
nization, has been researching and developing instruments to 
assess general education since 1976, when it received a Fund 
for the Improvement of Post Secondary Education (FIPSE) 
grant to develop a competency-based assessment of general 
education outcomes. As a result of that project, the College 
Outcome Measures Program was developed, and it served the 
needs of many institutions for almost two decades before it 
was withdrawn from use in the late 1990s. 

In 1986, ACT began development of a new assessment 
instrument, the Collegiate Assessment of Academic Profi- 
ciency or CAAP. ACT staff conducted an extensive review of 
relevant literature, convened a national advisory committee, 
and brought together committees of faculty experts in vari- 
ous subjects. A decision was made to develop tests in reading, 
writing skills, essay, mathematics, science (reasoning), and 
critical thinking. Test items were then developed by faculty 
content specialists from across the country. Once the initial 
development phase was completed, each item was thoroughly 
reviewed both internally and externally by faculty, bias, and 
measurement experts. CAAP, a norm-referenced test to as- 
sess student competency, was first piloted by more than 100 
institutions in 1988 and became operational in 1990. Several 
years later, a major study of student learning was undertaken 
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by the National Center of Teaching, Learning, and Assessment, headquartered at Penn 
State University. This study used CAAP participants at a representative sample of 
colleges and universities from across the nation to assess a number of general education 
outcomes. 

The CAAP tests are designed to measure the academic progress of students in gen- 
eral education. The purpose of CAAP is twofold. One, on a group basis, it is used to 
help institutions improve their instructional programs. Institutions looking at program 
evaluations can use the CAAP (a) to provide evidence that the general education objec- 
tives are being achieved, (b) to document change in student performance over time, and 
(c) to provide differential performance comparisons in general education instructional 
programs. Two, on an individual basis, CAAP may be used (a) to show readiness for 
further education, (b) to identify areas of interventions to assure student success, and (c) 
to document that a specified level of skill mastery has been attained prior to program or 
degree completion. 

Specifically, CAAP can be used to: 

1. Document achievement of selected general education objectives 

2. Indicate change from one educational level to another (i.e., "value-added" 
assessment) 

3. Compare local performance with that of other populations 

4. Establish requirements for eligibility to enter the junior year 

5. Establish other eligibility requirements, such as readiness to take specific 
advanced courses, to meet entry requirements into selected majors (e.g., nurs- 
ing, teaching, engineering, business), to meet requirements for entry into an 
upper-division college or university, or to determine graduation eligibility 

CAAP is used by hundreds of institutions (ACT, Inc., 2003a) each year to assess 
student learning for such purposes as program improvement, regional and professional 
accreditation, accountability, documenting student achievement, determining eligibility, 
and performance funding. 

Currently there are six equated forms of CAAP. Each test is 40-minutes in length 
(actual testing time) and may be administered independently or in combination. The 
CAAP allows maximum flexibility for institutions, so users can elect to test only those 
areas relevant to their needs. Each of the tests is described as follows. 

Reading 

The Reading Test consists of 36-items that measure reading comprehension as a 
combination of skills that cover two broad categories: referring (25-33% of the number of 
questions) and reasoning (67-75%). Referring skills require students to derive meaning 
from text by identifying and interpreting specific information that is stated explicitly. 
Students are asked to (a) recognize main ideas of paragraphs and passages, (b) identify 
important factual information, and (c) identify relationships among different components 
of the text. Reasoning skills require students to go beyond the information that is explicitly 
expressed. Items assess student ability to (a) determine meaning from context, (b) infer 
main ideas and relationships, (c) generalize and apply information beyond the immediate 
context, (d) draw conclusions, and (e) make comparisons. There are four passages, each 
with nine questions. A total score and two subscores are reported. 
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Writing Skills 

The Writing Skills Test consists of 72-items that measure students' understanding of 
the conventions of standard written English. Spelling, vocabulary, and rote recall of gram- 
mar rules are not tested. Two broad areas are addressed: (a) usage / mechanics (44%) and 
(b) rhetorical skills (56%). The usage / mechanics portion evaluates skills in punctuation, 
grammar, and sentence structure. The rhetorical skills section assesses strategy, organization, 
and style. The test consists of six prose passages covering a range of topics, each of which 
is accompanied by a set of 12 multiple-choice questions. A total score and two subscores 
are reported. 

Writing Essay 

The Essay Test is based on the assumption that the common writing skills taught in 
college-level writing courses include: 

• Formulating an assertion about a given issue 

• Supporting that assertion with evidence appropriate to the issue, position 
taken, and a given audience 

• Organizing and connecting major ideas 

• Expressing those ideas in clear, effective language 

Two 20-minute writing tasks, each defined by a short prompt that identifies a spe- 
cific hypothetical situation and audience, are presented. The situations involve an issue 
on which the examinee must take a stand and then explain why the position taken is the 
better alternative. A total score and a score for each essay are reported. 

Mathematics 

The Mathematics Test consists of 35-items designed to measure students' proficiency 
in mathematical reasoning. It assesses students' proficiency in solving mathematical prob- 
lems presented in many college-level mathematics courses and required in upper-division 
courses. It emphasizes quantitative reasoning rather than memorization of formulas. The 
content includes two categories: (a) basic algebra (49%), covering pre-algebra, elementary, 
and intermediate algebra; and (b) college algebra (51%) covering college algebra, coordinate 
geometry, and trigonometry. A total score and two subscores are provided. 

Science 



The Science Test has 45 items designed to measure students' knowledge and skills 
in the sciences. The test content is drawn from biological sciences, chemistry, physics, and 
physical sciences. It emphasizes scientific knowledge and reasoning skills rather than 
high-level mathematics or reading skills. The test consists of eight passages, each of which 
contains scientific information and a set of multiple-choice questions. Each passage uses 
at least one of the following formats: 

• Data representation (33%). Knowledge and skills measured include (a) graph 
reading; (b) interpretation of scatter plots; and (c) interpretation of informa- 
tion presented in tables, diagrams, and figures. 
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• Research summaries (54%). Students are provided with descriptions of one or 
more experiments and are asked to examine the design of the experiment and 
interpretation of the results. 

• Conflicting viewpoints (13% ). Students are presented with hypotheses or views 
that are mutually inconsistent owing to different premises, incomplete or 
disputed data, or differing interpretations of data. 

The items are classified as: understanding (18-22%), analyzing (49-53%), and general- 
izing (27-33%). There is a total score, but no subscores are reported. 

Critical Thinking 

The Critical Thinking Tests contains 32 items and measures students' skills in 
analyzing (53-66%), evaluating (16-28%), and extending arguments (19%). An argu- 
ment is defined as a sequence of statements that includes a claim that one of the state- 
ments, the conclusion, follows from the other statements. The Critical Thinking Test 
has four passages that are representative of the issues that are commonly encountered 
in a postsecondary curriculum. There is a total score, but no subscores are reported. 

CAAP multiple choice tests are scored on a linearly scale of 40 to 80, with a mean of 
60 and a standard deviation of approximately 5. Subscores, where relevant, are linearly 
scaled to have a mean of 15 and a standard deviation of 2.5. The essay test is scored on 
a six-point, modified-holistic scoring system. Two raters read each essay; and if they 
differ by a point, the chief scorer adjudicates and determines the reported score. 

Student motivation and demographic information is collected for each test to en- 
able a better analysis of student performance. Among the demographic data collected 
are: ethnicity, gender, age, English as a second language, year in school, student status, 
cumulative GPA, and educational plans. Reporting by individual demographic category 
is provided on the Institution Summary Report if at least 25 students are tested. 

CAAP provides two student score reports and, where students have performed 
at or above the national mean, a Certificate of Achievement. A detailed Institutional 
Summary Report and a Student Roster Report are provided at no charge. Additional 
services are available for a nominal fee, including a data diskette (contains data for 
institutions to carry out local research). Linkage Report (links ACT entry test data to 
CAAP), Combined Institutional Profile Service Report (combines data from several 
administrations), and customized reports. A new report, the Content Analysis Re- 
port, was introduced in early 2004. This report will provide an analysis of student 
performance on a group basis and will help institutions determine areas of program 
strengths and weaknesses. 

Unique to the CAAP is its ability to link back to entry assessment using the ACT 
Assessment, COMPASS, or ASSET. Institutions that use any of these other ACT tests 
as baseline data can document change in student achievement over time by compar- 
ing these data to CAAP performance. This service provides significant cost savings 
to institutions, and it eliminates the need to administer CAAP at entry, which allows 
institutions to follow their current entry protocols. The results of this approach can be 
seen in the Linkage Reports. For a longitudinal study, CAAP may be administered to 
entering first-year students to establish a baseline and then at the end of the sophomore 
year or the beginning of the junior year to document change from entry to completion 
of core coursework. 
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The Cognitive Level and Quality Writing Assessment Instrument 



eresa L. Flateby 



The Cognitive Level and Quality Writing Assessment 
(CLAQWA) instrument was designed at the University of South 
Florida to assist instructors with the assessment of the quality of 
students' writing and the cognitive levels demonstrated (Flateby 
& Metzger, 1999; 2001). We have discovered through application 
and investigation that CLAQWA is also effective for large-scale 
program assessment. Thus, instructors, administrators, and as- 
sessment professionals alike might benefit from a framework 
such as CLAQWA. 

CLAQWA was conceived approximately eight years ago 
in response to needs identified in a two-year general education 
learning community program in which writing was taught across 
the curriculum. Because multiple instructors were involved in 
the creation of the writing assignments and the evaluation of 
students' papers, a structure was needed to evaluate writing 
consistently. Accordingly, CLAQWA was designed to assist 
instructors and program evaluators with the assessment, di- 
agnosis, and grading of student writing and thinking. Faculty 
teams representing a diversity of departments, including English, 
measurement, first-year experience, and social sciences, were 
involved with the development and validation of the two-scale 
instrument. 

Designed to be flexible to accommodate instructors' needs, 
each scale can be used separately or can be combined to assess 
students' writing. The scale for cognitive level assessment ad- 
dresses the development of the writing prompt and the assess- 
ment of students' cognitive levels achieved in their writing. This 
scale, based upon the 1956 work of Bloom and his colleagues, 
is composed of four levels: (a) knowledge; (b) comprehension; 
(c) application; and (d) analysis, synthesis, and evaluation. (A 
portion of the Cognitive Level Scale is displayed in Figure 1. 

The quality of writing assessment scale consists of skills 
commonly found in writing texts (c.f., Hairston, Ruskiewicz, & 
Friend, 1999; Hodges, Horner, Webb, & Miller, 1998; Lunsford 
& Connors, 1992; Mulderig & Elsbree, 1990; Troyka, 1999) but is 
organized and clarified for any instructor who evaluates students' 
compositions. Three operationalized scale points are described 
for each subskill. For example, "The paper lacks a main idea," 
"Main idea is not clearly presented or is not maintained," and 
"The paper presents and maintains a main idea" are the descrip- 
tors for one subskill. (To view a sample of the Quality of Writing 
Scale, see Figure 2. 
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^Cognitive Level 



Indicate level of paper. Levels are 
arranged hierachially. Higher levels 
included characteristics of lower levels. 



Level 4: Analysis, Synthesis, Evaluation 
Makes a judgement based upon a given 
or constructed set of specific criteria, 
not opinion. Organizes or reorganizes 
ideas or combines elements to make 
a whole. Distinguishes between fact 
and fiction. Compares and contrasts 
or deduces. Identifies relationships of 
parts to the whole. 

Level 3: Application 
Uses what is understood in a new situa- 
tion. Uses what is learned 
in the assignment 
or in class. 

Level 2: Comprehension 
Translates or rephrases known words, 
interprets or explains in a way that 
demonstrates understanding of the 
material. 

Level 1: Knowledge 
Accurately recalls or describes, 
identifies information which was 
presented in class or reading. Involves 
memorization. 



High 

— Medium 

Low 

High 

— Medium 
Low 

High 

— Medium 
Low 
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— Medium 
Low 



Figure 1. Example of the Cognitive Level Portion of CLAQWA 



Because CLAQWA has important writing skills operationalized, encourages the conscious 
consideration of the cognitive level desired in students' writing, and helps develop confidence in 
an instructor's evaluation of students' writing, it has multiple functions. It is also flexible, permit- 
ting instructors to emphasize writing, cognition or a combination of the two in an assignment. It 
also is possible to target specific writing skills for a particular assignment or weight skills differ- 
ently. In short, the use of CLAQWA facilitates clear communication of instructors' expectations, 
encourages consistent assessment of students' papers, provides a framework for identifying and 
discussing strengths and weaknesses in papers, and facilitates feedback on students' writing. 
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ASSIGNMENT PARAMETERS 




Paper does not fulfill assignment 
requirements. 




Paper partially fulfills assignment 
requirements. 




Paper fulfills all the assignment 
requirements. 




The paper lacks a main idea. 




Main idea is not clearly presented. 




The paper presents and maintains 
a main idea. 




The appropriate audience(s) is (are) 
not addressed. 




The appropriate audience(s) is (are) 
not consistently addressed. 




The writer consistently addresses 
appropriate audience (s). 




The writer's purpose is not evident. 




The writer's purpose is not 
consistently evident. 




The writer's purpose is clear and 
specific. 


ORGANIZATION AND DEVELOPMENT: STRUCTURAL INTEGRITY 



Figure 2. Example of the Writing Skills Portion of CLAQWA 






The Cognitive Level and Quality Writing Assessment Instrument 
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CLAQWA has been used to develop a series of related assignments for a first-year 
experience course ("The University Experience") at the University of South Florida. In 
an effort to raise students' awareness of their role in becoming successful and responsible 
students and to guide them in an upward cognitive progression, three prompts were 
developed thematically and hierarchically. The first required students to write an essay 
describing an interview with a professor, which included student-constructed questions 
regarding his or her preferred teaching methods. The second asked students to research, 
describe, and explain three different teaching methods. The third required them to relate 
their preferred style of learning — which they identified in an earlier class — to the three 
teaching methods researched, determine which of the three is best suited to their preferred 
style, and describe how they could adapt to ensure success in classes less suited to their 
preferred style. With these assignments, it is hoped that students will gain a clearer under- 
standing of thinking levels and will develop strategies for adapting to different classroom 
environments. To view these assignments with two levels of specificity, see the appendix 
at the end of this essay. 

CLAQWA also provides instructors with a framework for evaluating students' pa- 
pers in the first-year experience course. Instruction on the CLAQWA instrument has been 
provided to assist these instructors — most without formal education in teaching methods 
or writing assessment — with the first-year experience programs' writing requirement. 
Several benefits result from this instruction. First, instructors learn to assess writing con- 
sistently. Second, when instructors explain the writing skills contained in CLAQWA, and 
those identified for a particular assignment, students have a clearer understanding of the 
instructor's expectations and just what constitutes quality writing. Third, when the in- 
structors present the cognitive scale, an awareness of thinking and learning beyond mere 
recognition and recall results. 

CLAQWA also is effective for assessing programs. At USF, students' writing and think- 
ing are studied within the broader assessment of general education. Students' writing is 
sampled when they first enter USF, at the completion of English Composition II, and during 
upper-level liberal arts courses. Because hundreds and even thousands of students' papers 
are evaluated, we originally used a holistic method for scoring. This approach is often the 
evaluation method of choice for large-scale assessment because of its efficiency and its success 
for achieving inter-rater reliability. This measurement criterion is of paramount importance for 
high-stakes, large-scale assessment, as is the case for entrance exams and statewide account- 
ability exams where a comparison to a particular group is desirable (Wolcott, 2000). Holistic 
scoring has been criticized, however, for not providing students with feedback regarding 
their strengths and weaknesses. Also, since the holistic method is often norm-referenced, it 
is less appropriate for measuring change over time and for identifying weaknesses which 
should be addressed. Typically with this method, anchor papers are selected from the pool 
of students' essays. These become the standard for assessing all papers within the pool. 
Thus, a "6" on a 1 to 6 scale could represent very different writing levels depending upon the 
type of classes or programs in which the writing assessment was completed, e.g., remedial, 
honors, first year, upper level. 

Because we wanted an efficient and consistent framework to evaluate students' writ- 
ing at several points in students' undergraduate programs and one that would provide our 
students with some feedback on the strengths and weaknesses of their papers, we collapsed 
the CLAQWA analytic scale into five major categories. We then compared holistic scores 
and scores from this abbreviated CLAQWA scale for multiple sets of papers. These studies 
revealed substantial and respectable correlations (e.g., .80 and above). Because we needed 
an assessment tool that would provide a constant set of criteria for evaluating change in writ- 
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ing and a method for identifying problem areas and because the results from the collapsed 
CLAQWA scale were favorable, we substituted the holistic scale with the abbreviated 
CLAQWA scale. More recently, we have scored students' papers with the comprehensive 
CLAQWA scale to identify greater detail regarding students' performance. 

In conclusion, CLAQWA appears to serve multiple functions: (a) establishing criteria 
for specific assignments, (b) communicating the writing and thinking skills expected for 
an assignment, (c) providing guidance for consistently assessing papers, and (d) offering 
a framework with which to discuss papers' strengths and weaknesses. CLAQWA also 
is effective for assisting with program assessment by providing a mechanism for change 
over time, identifying specific strengths or weaknesses in students' writing, and assessing 
achievement of a specific writing skill level. 
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Appendix 

University Experience Writing Assignments 



Assignment 


Sample Assignment A 


Sample Assignment B 


Faculty Interview 


Describe one of your 

professor's answers to the 

following questions: 

1. Name, faculty rank, and 
highest academic degree 
earned and their discipline 
of study 

2. Length of time at USF and 
other universities before 
USF 

3. Where and when highest 
degree was earned 

4. Undergraduate 
institution(s) they attended 

5. Their biggest problem as 
an undergraduate and as a 
graduate student 

6. Classes they wouldn't have 
taken — knowing what they 
know now about what 

it takes to succeed as a 
student 

7. The method of teaching do 
they use most often and 
why they use it 

8. Advice they would give 
students to succeed at USF 

9. The answer to at least 
one question of your 
choice — something you've 
always wanted to know 
about teaching, the faculty 
member you chose to 
interview, or another 
pertinent question 


You have been asked to 
write an article for the Oracle 
about a USF instructor 
whose course is causing you 
difficulty. The instructor 
or the content could be the 
source of your difficulty. 
Develop interview questions 
with your classmates to ask 
your instructor and include 
at least one question about 
the instructor's favorite 
teaching methods. 

Your article should be ap- 
proximately two typewritten 
pages and should describe 
the interview, based upon the 
responses to the questions. 
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Assignment 


Sample Assignment A 


Sample Assignment B 


Teaching Methods 


Describe and explain at least 
three teaching styles 


You have been invited to 
write an essay to prospective 
teachers describing three 
teaching methods. You 
will need to research 
three methods, which 
may differ with regard 
to characteristics such as 
instructors' roles, students' 
roles, environmental setting, 
and students' ages. As you 
conduct your research and 
write your essay, determine 
which learning style is suited 
to each method and include 
this in your essay. 

The writing project for this 
assignment will be a three- to 
five-page essay that informs 
prospective teachers about 
your findings. Your essay 
should incorporate research 
from at least five sources and 
follow APA style. 


Learning Style 


Of the teaching styles 
described and explained in 
the above assignment, which 
style is best for your particular 
learning style? Give your 
rationale. 


Write a two-page essay to 
me in which you identify the 
teaching method researched 
in the previous assignment 
which least suits you learning 
style. What do you now think 
you could do to improve 
your chances of success when 
that method is used? Which 
of the three methods is best 
suited to your learning style 
and why? Make sure you 
identify to me what your 
preferred learning style and 
describe its characteristics. 
Feel comfortable using "I" in 
the essay. 
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The Hope Scale: 
Willpower ai 
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Measurement of 
Waypower 

A few years ago, my college president handed me the 
weighty copy of Charles R. Snyder's Psychology of Hope (1994). 
Tucked away in this study is the fulcrum of much of Snyder's re- 
search — a simple, yet remarkable tool he terms "The Hope Scale." 
Hope, according to Snyder, is a thinking process in which people 
have a sense of agency (willpower) and pathways (waypower) 
for goals. 

Snyder's work is intriguing to me, because much of my 
recent research has been in the area of student motivation, espe- 
cially intrinsic motivation. I have delivered 20 or so workshops 
around the topic at regional and national conferences. Several of 
these sessions were entitled, "Student Success or Student Non- 
Dissatisfaction?" (Pattengale, 2003b). In short, I found through 
surveying the above conference audiences that the overwhelming 
majority of retention and student success programs were founded 
on the premise of removing areas causing student dissatisfaction. 
In other work, I ask if some universities should change the sign 
above from "Office of Student Success" to "Office of Student Non- 
Dissatisfaction." (Pattengale, 2003a). While a focus on satisfaction 
usually proves helpful and often correlates with improved reten- 
tion, it does not get at the core of the student experience. It focuses 
on the "how" rather than the "why." The latter, in my opinion, 
should be the primary concern of student success programs. Thus, 
Snyder's preoccupation with "Willpower" and "Waypower" 
proved provocative. Though he stops short of "Why" questions 
and assumes students are goal oriented, his work remains helpful 
in understanding topics related to a student's core. 



The Hope Scale 

The Hope Scale has only 12 questions, much shorter than 
most lengthy student surveys. The following are the eight ques- 
tions actually measured. 

1 . I can think of many ways to get out of a jam. (Path- 
ways) 

2. I energetically pursue my goals. (Agency) 

3. There are lots of ways around any problem. (Path- 
ways) 

4. I can think of many ways to get the things in life that 
are most important to me. (Pathways) 
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5. Even when others get discouraged, I know I can find a way to solve the problem. 
(Pathways) 

6. My past experiences have prepared me well for my future. (Agency) 

7. I've been pretty successful in life. (Agency) 

8. I meet the goals that I set for myself. (Agency) 

Students rate these statements from 1 (definitely false) to 4 (definitely true). The sum 
of these answers provides the Hope Score. The designation of questions is in parentheses. 
Questions related to willpower are labeled "agency," and those related to waypower are 
labeled "pathways." 

My research colleague, Michael Boivin, agreed to test the tool over a two-year period 
with more than 1,000 students. Under his mentorship, Heidi Ihrke (2002), at the time a 
student at Indiana Wesleyan, conducted a key aspect of the study. She concluded that 
"Hope as a measurable construct is an important component of likelihood of student 
success in facing the challenges of transition to the university learning community in 
the freshman year, and is relevant to spiritual and emotional well-being in that transi- 
tion process" (p. 1). The Hope Scale was administered in conjunction with the College 
Student Inventory (CSI) and the Spiritual Well-Being Inventory (SWB) to nearly 600 first- 
year students and was significantly related to student success indicators on the CSI and 
existential well-being indicators on the SWB. A group of sophomore-level psychology 
students took the Hope Scale in conjunction with the 16-Personality Factor Inventory 
(16-PF) for assessing personality traits. Here, the Hope Scale was found to be related to 
measures of emotional well-being and leadership styles (Ihrke). 

In 2001, Boivin and graduate student Ellen Jones conducted a separate study with a 
different cohort to examine relationship between the Hope Scale and critical dimensions 
of personality. Some of these results are influencing our decisions in our new Center for 
Life Calling and Leadership (made possible through the generous funding of The Lilly 
Endowment). 

The Hope Scale will help us design student success tasks, especially when the evi- 
dence seems to support the notion that the stronger the hope of fulfilling a dream, the 
more likely a college student will remain in school. Snyder and his colleagues at the 
University of Kansas-Lawrence have not only discovered ways to measure hope, they 
have also found strong correlations between one's belief in their abilities to reach goals 
and goal attainment. According to Snyder (1996), "high-hope individuals" typically: 

1. Clearly conceptualize their goals 

2. Envision one major pathway to a desired goal and can generate alternative 
pathways, especially when the original one is blocked 

3. Perceive that they will actively employ pathways in pursuit of their goals 

In my curriculum on motivation, I use the actions of heroes to help students un- 
derstand the dynamics of their passion and direction. Likewise, Snyder studied what 
high-hope people say and do. Based on these observations, and on his research on the 
"two necessary components" of "goal directed thinking" — agency [willpower] and path- 
ways [waypower]" — he has provided dozens of careful studies of motivation that have 
applications for student success studies. (Curry, Snyder, Cook, Ruby, & Rent, 1997, p. 
1257) Snyder notes: "Although pathways and agency thinking are two distinct compo- 
nents of the hope model, they are functionally inseparable. In fact, they are theorized to 
influence one another reciprocally, such that a change in one will cause a commensurate 
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change in the other" (Snyder, 2002, p. 299). Snyder gives the following suggestions for 
enhancing hope: 

• Learning self-talk about succeeding 

• Thinking of difficulties encountered as reflecting wrong strategy, not lack of 
talent 

• Thinking of goals and setbacks as challenges, not failures 

• Recalling past successes 

• Hearing stories of how other people have succeeded (e.g., movies, tapes, 
books) 

• Cultivating friends with whom you can talk about goals 

• Finding role modes that you can emulate (everyday heroes are closer than 
you think) 

• Exercising physically (relearning that the body and mind are connected) 

• Eating properly (remembering that you need fuel) 

• Resting adequately (recharging for the next active goal-directed output) 

• Laughing at oneself (especially when stuck) 

• Regoaling (persistence in the face of absolute goal blockage deflates agency 
and pathways) 

• Rewarding oneself for small subgoal attainments on the way to larger, long- 
term goals 

• Educating oneself for specific skills, as well as learning how to learn 

Snyder (1995) also notes that the common process among interventions to promote 
positive change or growth is increasing feelings of agency and opportunities or "pathways" 
for achieving personal goals. 

Colleges spend considerable resources to determine which students are most likely to 
succeed. Thus, a tool like the Hope Scale can prove valuable in the admission process. Also, 
it can help in targeting students who are at-risk due to low motivation. Several scholars 
note that one question more than any other seemed to identify at-risk students-"When the 
going gets tough, I . . .?" Snyder's work sheds light on this response. 

While a full discussion of Snyder's theories is beyond the scope of this article, his 
various publications of the Hope Scale are ripe for further study and pregnant with applica- 
tion (e.g., http: / / www.ku.edu/ -crsnyder / child.htm). He gives us a glimpse of students' 
pursuit of goals and how they handle challenges. This scale is worthy of a closer look. It 
might save a university considerable funds and hours of administrative oversight. More 
important, it might edge first-year programs closer to looking at life goals. Snyder stops 
short of this important step, but points us in the right direction. Perhaps we could add the 
category of "wantpower" (Pattengale, in press). 
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Styles? Can We Identify Them? What is Their 
ssessment Program? 



Linda Suskie 



I had an epiphany of sorts a few years ago when I began 
attending a different place of worship. At my prior place of wor- 
ship, when a speaker read passages aloud at service, I followed 
along in a booklet. My new place of worship did not provide 
such a booklet, and without the written backup, I was struck by 
how difficult it was for me to understand what was being said. 

I realized from this experience that I do not learn well by 
listening and that I had subconsciously developed adaptive 
strategies over the years, such as following along in a book. If I 
do not learn well by listening, how do I learn best? While I can 
learn by reading, I realized that I learn best by visualizing: us- 
ing tools such as maps, outlines, Venn diagrams, flow charts, or 
bulleted lists. 

My experience in uncovering my learning preferences con- 
vinced me that the concept of learning style makes intuitive sense. 
But does empirical research bear out my intuition? Are there 
valid models of learning style? Can learning styles be identified 
reasonably accurately? And, most important, can information 
on learning styles help faculty and assessment practitioners give 
students the best possible learning experience? This article will 
attempt to address these questions. 



What is "Learning Style"? 



A frequently cited definition of learning style is "cognitive, 
affective, and physiological traits that serve as relatively stable 
indicators of how learners perceive, interact with, and respond 
to the learning environment" (Keefe, 1982, p. 44). Beyond this 
general definition, however, conceptualizations of learning style 
vary widely, as the models discussed in this article make clear. 



Do Learning Styles Exist? 

Most educators "acknowledge the existence of learning 
styles and their impact on learning process" (Vincent & Ross, 
2001). Some people, for example, clearly prefer reading books to 
listening to them on tape, and some people clearly prefer work- 
ing alone to working with others (Curry, 1987). Indeed, some 
learning preferences, such as favoring a quiet background, seem 
so self-evident that a validated instrument may not be necessary 
to assess them. As Nagy (1995) notes, "Little can be said about 
preference questions that ask, for example, what time of day a 
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student prefers to study, except to wonder if such information requires the expense of a 
standardized test" (p. 567). 

Learning style advocates point to a number of studies validating the existence of 
learning styles. Swanson (1995), for example, cites several studies identifying cultural 
differences in learning style. Critics assert, however, that for a learning style theory to be 
valid and useful, researchers must show that students learn more effectively when their 
learning styles are accommodated than when they are not, and only a limited number of 
studies have shown this. Some critics, therefore, feel that the usefulness or validity of learn- 
ing style models and instruments has not been definitively established (Bonham, 1988a; 
Bonham, 1988b; Kavale & Forness, 1987; Rayner & Riding, 1997). 

A particular concern (Grasha, 1990; McKeachie, 1995; Stellwagen, 2001) is that most 
learning style theories label or pigeonhole students into a few discrete, quantitative, and 
often dichotomous categories rather than recognizing that learning style evolves as one 
learns and grows and varies by discipline (Silver, Strong, & Perini, 1997). 

What Models and Instruments Are Available to Identify Learning Styles? 

Learning style models abound, as do myriad published and web-based instruments. 
The models are not mutually exclusive or necessarily complementary (Vincent & Ross, 
2001). Here are some of the most prevalent models. 

Field Dependence/Field Independence Model 

Field dependent learners pay more attention to what they see than what they feel. 
Their perceptions are influenced by their environment; they use their entire surroundings to 
process information, and they focus on the whole rather than on parts. They are externally 
motivated, prefer to work collaboratively rather than independently, are people-oriented, 
and are affected by instructor's interaction and communication style. 

Field independent learners, meanwhile, pay more attention to what they feel than 
what they see. Their perceptions are not influenced by their environment; they focus on the 
parts rather than the whole. They are intrinsically motivated, prefer to work independently 
or competitively, and take a more impersonal approach to learning. 

The field dependence/ field independence model has successfully predicted academic 
performance in a number of studies, which suggests that field independent students are 
more likely than field dependent students to succeed academically (Hayes & Allinson, 
1997; Thompson et al, 1979; Wilson, 1998). Instruments based on the field dependence/ field 
independence model include the Group Embedded Figures Test (Witkin, 1971) and the 
earlier Embedded Figures Test (Benton & Spreen, 1969). 

Jungian Models 

Jungian models are based on the work of Carl Jung. The best-known application of 
his work is the Myers-Briggs Type Indicator (Briggs & Myers, 1998), which identifies four 
personality dimensions: (a) Extraversion-Introversion, (b) Sensing-Intuition (preferring the 
concrete or the abstract), (c) Thinking-Feeling (preferring logic or values), and (d) Judging- 
Perceiving (being organized or flexible and easygoing). 

Other instruments using elements of Jungian models include the Gregorc Style De- 
lineator (Gregorc, 1985), the Keirsey Temperament Sorter II (Keirsey, 1998), and the Kolb 
Learning Style Inventory (Kolb, 1999). Opinions of the validity of the Gregorc and the Kolb 
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are mixed (see, for example, Seidel & England, 1997; Sewall, 1986 regarding the Gregorc 
and Hayes & Allinson, 1997; Curry, 1987 regarding the Kolb). There appears to be limited 
evidence of the validity of the Keirsey (see, for example, Keirsey, 1998; Kelly & Jugovic, 
2001 ). 

Sensory Models 

Under these models, visual learners prefer to learn through visual stimuli such as 
graphs and charts; auditory or aural learners prefer to learn by listening; and tactile, haptic, 
or kinesthetic learners prefer to learn through hands-on experiences involving touch or 
bodily movement. Some models identify additional sensory preferences. 

Instruments based on sensory models include the DVC Learning Style Survey for College 
(Miller, 2000), the Perceptual Modality Preference Survey (Cherry, 1997), the Personal Learn- 
ing Style Inventory (Wyman, 1999), the Self Administered Inventory of Learning Strengths 
(Siegel & Lester, 1994), the VARK Questionnaire (Pleming, 1998), and What's Your Learning 
Style? (Bogod, 2002). All these instruments appear to have little or no evidence of reliability 
or validity. Because they all address sensory perceptions; however, they make intuitive sense 
and may have some face validity. 

Social Interaction Model 

Under this model, collaborative or cooperative learners enjoy working harmoniously 
with peers, competitive learners are suspicious of peers, and individualist learners prefer 
learning in isolation. One instrument based on this model is the Learning Preference Scales 
(Owens & Barnes, 1992), but it has limited evidence of reliability and validity (Perro, 1998; 
Miller, 1998). 

Howard Gardner's Multiple Intelligences Model 

Gardner has identified eight intelligences (1983; Checkley, 1997): (a) logical / mathematical, 
(b) visual/ spatial, (c) bodily /kinesthetic, (d) musical, (e) linguistic, (f) interpersonal, (g) in- 
trapersonal, and (h) naturalist. Widely used in basic (K-12) education. Silver, Strong, and 
Perini (1997) note that the model is "backed by a rich research base" (p. 23). Others assert, 
however, that few studies validate Gardner's model (Klein, 1997; Traub, 1998). 

Gardner has not developed an instrument to assess multiple intelligences and, indeed, 
opposes the use of multiple-intelligence "tests" for a number of reasons (1996). Instruments 
based on his model include the Multiple Intelligence Inventory (Learning Disabilities Re- 
source Community, 1983), Multiple Intelligences Developmental Assessment Scales (Shearer, 
1999), Multiple Intelligences Inventory for Adults (Armstrong, 1999), and Test Yourself — How 
Are You Smart? (Rose & Nicholl, 1998). It appears that none of these instruments has been 
effectively validated. 

Biggs' Approaches to Learning Model 

Under this model, students with a surface approach to learning aim to meet minimal 
standards and learn unintegrated details, while students with a deep approach have an intrin- 
sic interest in learning and learn meaning and relationships. Biggs' Study Process Question- 
naire (1987) and revised two-factor Study Process Questionnaire (Biggs, Kember, & Leung, 
2001) have evidence of validity (Brown, 1992; Hall, 1992; Sachs, Law, & Chan, 2003). 
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Multiple Models 

Some learning style instruments appear to combine several models. The Index of 
Learning Styles (Soloman & Felder, 1987), for example, assesses four learning style di- 
mensions: 

1. Active (prefers to do something active with information, such as discuss, 
apply, or explain) - Reflective (prefers to think about it) 

2. Sensing (prefers to learn facts and follow established processes) - Intuitive 
(prefers abstractions and learning by discovery) 

3. Visual (prefers to learn through pictures, diagrams, and demonstrations) 

- Verbal (prefers to learn through written and spoken words) 

4. Sequential (prefers to learn in linear steps) - Global (prefers to absorb mate- 
rial almost randomly and put things together in novel ways) 

The authors acknowledge that "the ILS is still under development and cannot be consid- 
ered as having been validated" (Index of Learning Styles, n.d.). 

The Productivity Environmental Preference Survey (Dunn, Dunn, & Price, 1993) as- 
sesses a multitude of dimensions, including: 

1 . Immediate Environment (with subscales for noise level, light, temperature, 
design, emotionality, motivation, persistence, responsibility, and structure) 

2. Sociological Needs (with subscales for learning alone / peer oriented, author- 
ity figures present, and learn in several ways) 

3. Physical Needs (with subscales for auditory, visual, tactile, kinesthetic, requires 
intake, evening-morning /late morning/ afternoon, and needs mobility) 

This model has been used in countless studies, and some feel that it has been well 
validated (Curry, 1987; Dunn & Griggs, 1995; Lewthwaite & Dunham, 1999). Others, 
however, strongly criticize the model as being unvalidated (Bonham, 1988a; Kaiser, 1998; 
Kavale, Hirschoren, & Forness, 1998). 

The Learning and Study Strategies Inventory (Weinstein, Palmer, & Schulte, 2000) 
focuses on behaviors rather than temperaments or predispositions. Subscales examine 
motivation, anxiety, time management, attitude, concentration, information processing, 
selecting main ideas, and test taking. A more detailed overview of LASSI is available 
elsewhere in the monograph. Opinions of its validity and value are mixed (Blackwell, 
1992; Hayes, 1992). 



Can Learning Styles Be Meaningfully Identified? 

Unfortunately, few learning style models and instruments have been well validated 
(McKeachie, 1995), so it is questionable whether most existing instruments can accurately 
identify a student's learning style. But just as quizzes in popular magazines can some- 
times give us useful insight into ourselves, unvalidated learning style instruments may 
give our students constructive clues about how they learn best. The key is not to take the 
results of any one instrument too seriously. Students should not let any one instrument 
dictate their learning style (Bonham, 1988a), and results should not be used to make any 
significant, potentially harmful decisions about students, pedagogy, the curriculum, and 
other areas. 
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How Might Students Identify Their Learning Styles? 

As Guild (1997) notes, learning is a complex process, and students learn in a broad 
variety of ways. Because any learning style instrument is imperfect and assesses only cer- 
tain learning style dimensions, any one learning style instrument will give an incomplete 
picture. It is therefore sensible to ask students to take the initiative in reflecting on their own 
learning styles, using several instruments, not just one, to increase their self-awareness. 



• Ask students to write a paragraph explaining how they learn best (Crowe, 
2000; Grasha, 1990). 

• Have students complete at least two learning style inventories that reflect 
different models, and compare the results with their self-description. 

• Ask students to reflect on what and how they learned from a recent 
assignment. 

• Have students use all of this information to refine their statement of how 
they learn best. Their statement should include study/ learning strategies 
they might use to best take advantage of their particular combination of 
learning styles and to help them learn in situations where they must use 
approaches that do not correspond with their style. 



How Can Faculty Use Information on Learning Styles to Help Students Learn? 

Perhaps the primary benefit of information on learning styles is helping faculty un- 
derstand better their students and how they learn (Guild, 1997). Faculty should first un- 
derstand that students might use learning styles that may not match their own. Schroeder 
(1993) reports that more than 75% of faculty learn best through abstract concepts, ideas, 
and theories, compared to just 40% of entering students and 25% of the general popula- 
tion. Most students, in contrast, learn best through concrete, practical, structured, and 
sequential experiences. 

With this knowledge, faculty can help students by using a variety of teaching / learning 
strategies so that all students, regardless of learning style, have at least occasional opportu- 
nities to learn in settings congruent with their preferred styles (Anderson & Adams, 1992; 
Wilson, 1998). Specific suggestions for doing this come from a variety of authors: Clark 
(2000), Montgomery & Groat (2002), Ogden (2003), Siegel and Lester (1994), and Vincent and 
Ross (2001). In addition to the usual lectures and readings, for example, engage students' 
senses and give them the structure that some prefer by providing visual aids such as lists 
or diagrams, written outlines of key points, structured opportunities for group interaction, 
practical "real world" examples, and a variety of assignment formats. Also try to get to 
know your students, help them get to know you, and provide plenty of feedback. Finally, 
include curricular experiences, such as the group of exercises described above, that help 
students learn how to learn (Claxton & Murrell, 1987). This is particularly apropos for 
faculty teaching in first-year experience programs. Most good teachers, of course, already 
do these things instinctively. 

Faculty should not try to customize their teaching to accommodate every student's 
individual learning style, as learning style models are not yet sufficiently validated to be 
able to determine definitively how each student learns most effectively (Zarghani, 1988). 
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Furthermore, while students should use their "strong" learning styles to best advantage, it 
is just as appropriate for them to develop their abilities to use other learning styles (Grasha, 
1990; McKeachie, 1995; Montgomery & Groat, 2002) and to work with faculty whose styles 
differ from their own. Finally, as Gardner (1996) states, "There is no point in assuming that 
every topic can be effectively approached in at least seven ways, and it is a waste of effort 
and time to attempt to do this" (p. 17). 

How Might Learning Styles Be Part of an Assessment Program? 

Many of us focus our assessment efforts on the outcomes of a course or program, but it is 
also useful to assess learning processes, contexts, and inputs, as these factors help us understand 
the outcomes. Learning style is an input, along with high school record and other attributes 
that students bring with them to college. Information on learning style can be helpful when 
assessing learning processes or outcomes; it may help explain why some teaching and learn- 
ing strategies are not consistently effective. 

One of the most exciting applications of learning styles to assessment is classroom research 
that investigates how information on learning styles can improve teaching (Claxton & Murrell, 
1987). It is clear that we desperately need more research on learning styles: what they are, how 
do we assess them, and how do we use them to help our students make the most of their learn- 
ing opportunities. Anything we can learn about learning styles through our own assessment 
efforts will help not only our students but our colleagues throughout academe as well. 

Author's Note 

The author gratefully acknowledges the contributions of Alexis Turrentine, former 
graduate assistant. Office of Assessment, Towson University, to this article. 
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Assessing the Firs 



Tracy L. Skipper & 
Marla Mamrick 



College Year: Some Concluding Thoughts 



As the essays in this monograph make clear, there are a 
variety of approaches for conducting assessment on our cam- 
puses. Given the critical importance of the first college year, it 
seems obvious that we would focus our assessment efforts on the 
programs, courses, curricular structures, and services designed 
to help students make the transition to college and succeed on 
our campuses. In this concluding essay, we attempt to synthesize 
some of the larger themes seen throughout the monograph. 



Reasons to Conduct Assessment 



Many reasons exist to conduct assessment. This monograph 
highlights a broad array of methods and describes the outcomes 
achieved; however, we need to know much more than this. When 
performing assessment, we must also know why the informa- 
tion is needed and how we are going to use the findings to bring 
about positive change. Reasons for conducting assessment vary 
and can include the following: 



• Responding to demands for accreditation or accountabil- 
ity. Accrediting bodies set criteria that institutions, 
departments, and programs must meet in order to 
be deemed well-functioning and offering the level of 
knowledge and service needed for student success. 
Assessment is an easy way to demonstrate that the 
key criteria are being addressed and standards are be- 
ing met. In addition to accrediting bodies, institutions 
must respond to the demands of external funders 
and governing boards to demonstrate that students 
are performing at an acceptable level and achieving 
the learning outcomes established by the institution. 
Demands for accountability also come from internal 
constituencies, as department chairs and program 
administrators must demonstrate the effectiveness of 
curricular structures, courses, programs, and services 
to compete for increasingly scarce resources. In all 
cases, assessment can be used to demonstrate how 
a program benefits students and the institution and 
to show how the program is meeting expectations 
and contributing to established goals. 

• Demonstrating student persistence and success. Perhaps 
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the number one outcome individuals and institutions point to when responding 
to demands for accountability in first-year initiatives is retention — we want 
to know whether the program in question leads to continuous enrollment 
for students and ultimately, to graduation. Assessment efforts can certainly 
report persistence and graduation rates and the academic success of students, 
but they can also help us determine why programs are contributing to student 
success and which students are most likely to benefit. Assessment results can 
also help us understand why programs, courses, or services are not producing 
the level success and persistence that we might expect. When programs are 
successful, assessment may help us explain why different groups experience 
different outcomes. 

• Piloting new programs. The Freshman Absence-Based Initiative at the University 
of Mississippi is a good example of using assessment during the pilot phase. 

Before committing resources to an untested, large-scale initiative, adminis- 
trators frequently want to know "will it work?" Building an assessment plan 
into the pilot allows program designers to report on whether or not the pro- 
gram is meeting the established goals. The results of assessment also help us 
understand what changes need to addressed before undertaking large-scale 
implementation. It is necessary to assess all aspects of the program, from goals 
to outcomes, to determine whether or not it is worthwhile to continue. 

• Responding to natural curiosity. While assessment is clearly tied to issues of 
program maintenance and improvement, we may choose to engage in assess- 
ment simply to learn more about our work. Assessment efforts can help fulfill 
our desire to know more about what we are doing and the impact it is having 
on others. When our initial assessment efforts generate unexpected results, 
we may also be curious to know why that particular set of results appeared. 

Many of the reasons institutions chose to do assessment focus on an examination of 
the end results, but we should never engage in assessment activities to merely confirm 
existing beliefs. Even when we discover what we believed to be true, we should strive to 
push past it in order to understand the why and how of the outcomes. At the same time, 
we must be open to the questions that assessment uncovers and be prepared to view this 
effort as an evolving, ongoing process rather than a one-time-only activity. 

Benefits of Assessment 

The benefits of an effective assessment program seem fairly obvious: Assessment 
offers the opportunity for program justification and improvement. When assessment sup- 
ports effective initiatives for students, it may also translate into improved retention and 
persistence, positive learning outcomes, and satisfaction with college. Yet the benefits of 
assessment efforts go beyond the bottom line, as authors in this volume make clear. Suc- 
cessful assessment initiatives can produce the following results: 

• Productive campus dialogue. Assessment efforts frequently generate campus 
dialogue around issues that are important to students but that have not been 
previously voiced or seen as a concern. The Prompts Project at Virginia Com- 
monwealth University raised awareness of important, but unspoken issues as 
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faculty and administrators reviewed student responses to weekly questions 
and began to bring these responses into campus meetings. As such, these 
dialogues can create the energy needed for true campus transformation. 

• Greater buy-in and ownership of courses and programs. Repeatedly, we have seen 
two elements in successful assessment efforts. The first is the involvement of 
a wide range of stakeholders in the development of the assessment program, 
and the second is the wide dissemination of assessment results. The first ele- 
ment ensures that assessment results will be seen as valid, while the second 
can increase the visibility and regard of a program or course on campus. Par- 
ticipants in assessment — whether as members of the planning team or as those 
who participate in the implementation plan — may themselves experience a 
greater sense of buy-in or ownership for the programs that they are assessing 
or of which they are a part. For example. Debora Scheffel and Marie Revak 
report that the process of keeping course diaries increased faculty members' 
understanding of the value and importance of the first-year seminar. It also 
appeared to increase the likelihood that they would teach the course again. 

Thus, faculty participation in assessment at the grassroots level has the po- 
tential to transform their attitudes toward programs and courses. 

• Increased innovation. Closely related to the previous point, is the role that as- 
sessment plays in innovation. As Randy Swing points out in his overview of 
the monograph, assessment should not be done merely for assessment's sake. 
Underlying these efforts must be the notion of program improvement. Par- 
ticipation in assessment can increase our curiosity and willingness to experi- 
ment, especially if we have faith in the results of assessment. For example, if 
X does not appear to be delivering the desired result in this class, or program, 
or with this group of students, faculty or program administrators may ask 
such questions as: What can I do to change that result? Flow do I transform 
the class or program? What additional information do I need to know about 
this group of students to be able to serve them better? When well-designed 
assessments offer answers to these questions, faculty and administrators can 
generate effective plans for change. 

Choosing an Assessment Model 

The essays in this volume are not intended to present readers with a blueprint for 
conducting assessment. Rather, they offer a range of options from which to choose and 
which must necessarily be tweaked to meet the needs of any particular campus. These es- 
says do, however, provide us with some good information about the advantages and dis- 
advantages of different assessment methods. A good portion of the monograph is devoted 
to standardized, commercially available assessment instruments. Many of these offer the 
possibility for institution-specific questions, which allow campuses to tailor surveys to their 
unique assessment needs. Such surveys are typically easy to administer, and the quantita- 
tive results they produce are easy to compile and report, but the picture they present may 
be incomplete. 

Qualitative assessment methods — including, interviews, focus groups, and text 
analyses — offer potentially richer data than quantitative methods and may be the only way 
to analyze certain pheonomena. For example, a wide range of cognitive tests exist to help 
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us pinpoint students' intellectual development, but these tests give very little insight into 
the cognitive processes of students' intellectual life. The think-aloud protocol, described by 
Lendol Calder and Sarah-Eva Carlson, allows us to see how students think or make mean- 
ing — something a test of cognition cannot do. Qualitative methods are frequently more 
time consuming, more difficult to implement, and more difficult to analyze than quantita- 
tive assessment strategies. Yet they are valuable, because they help us answer questions 
not addressed through survey research or to explain why certain outcomes are reported 
through quantitative assessments. 

The standardized assessments reviewed here focus on a narrow range of experi- 
ences, outcomes, or behaviors. As noted above, some may be customized with the addition 
of institution-specific questions. On the other hand, the qualitative methods explored here 
can be adapted to a wide range of situations beyond those described in these essays. The 
flexibility of such methods is exciting. 

Both quantitative and qualitative approaches have a place in the assessment arsenal, 
and institutions and program and course administrators would be wise to use a carefully 
selected set of both types when developing assessment programs. 

Moving Forward with Assessment 

The essays collected here introduce readers to many valuable survey instruments 
that can be used to investigate critical issues in the first college year. They also highlight 
campus-specific approaches that can be adapted widely. In no way, is this collection intended 
to serve as the last word on assessment in the first year of college. We continue to depend 
on practitioners to develop and refine assessment methodology and to share their work 
with others. This continual focus on assessment of the first college year can only serve to 
improve the learning and success of entering students on all our campuses. 



Typology of Instruments for Assessment of the First College Year 

Randy L. Swing 



These surveys are administered in high school, during the college admissions process, 
or during new student orientation. Survey participants report their expectations, impres- 
sions, goals, and/ or hopes for the college experience or they report their pre-enrollment 
behaviors and experiences. These surveys 

• Provide baseline data, telling us who our students are at the point of entry 

• Form gain scores when matched with posttests 

• Provide covariates and controls for advanced statistical evaluations 



These surveys are designed for use near the end of the first college year and collect 
a multi-perspective report about collegiate experiences. There are only three national 
instruments designed just for use at the end of the first year, and each has been developed 
within the last five years as part of The Pew Charitable Trusts' accountability agenda 
for higher education. The intended timing of these surveys is their common link, but 
otherwise the three instruments are very different in that one is designed just for use in 
community colleges, one only for use at four-year institutions, and one can be used at 
two- or four- year institutions. Two of the instruments (one for two-year institutions 
and one for four-year institutions) focus on student engagement with higher education 
in terms of how students spend their time. The other instrument collects a wide array 
of student opinions, beliefs, and behaviors. One instrument is designed to assess both 
first-year and senior students, another is designed as a post-test when combined with a 
matching survey at the college entry point, and the third survey is not specifically de- 
signed as a repeat measure. 



These surveys take a holistic approach by collecting information on a variety of col- 
lege experiences and environments. Examples of topics include 

• "Average time" spent on academic and co-curricular tasks 

• Frequency of contact with peers, faculty, and staff 

• Self-reported gains in knowledge and self-confidence 

• Study skills such as time management or note-taking 

• Satisfaction with college 



\ 




Types of Instruments 
Pre-Enrollment/Baseline Data 



Surveys of Outcomes and Experiences in the First College Year 



General Surveys of Student Behavior, Attitudes, 
Study Skills, Satisfaction, & Experiences 
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• Alcohol /drug use 

• Life management skills (e.g., relationships with roommate, parents, partners) 

Surveys of Specific Services/Units/Programs 

These surveys investigate a particular slice of the college experience with a series of 
narrowly drawn and specific questions about the full range of a given service, unit, or pro- 
gram. Instruments may include demographic and self-report questions so that opinions 
can be disaggregated. Examples of available instruments include those focused on: 

• Academic advising 

• Residence life 

• Campus student unions 

• First-year seminars 



Surveys of Specific Populations 

Surveys in this group primarily provide information to evaluate the experiences, sat- 
isfaction, or behaviors of a specific group of students. The focus is on understanding the 
specific group, cohort, or sub-population. Examples of sub-populations include: 

• Adult learners 

• Fraternity or sorority members 

• Non-returning students 

Placement and Academic Knowledge Surveys/Tests 

These instruments are designed to test academic knowledge and skills. Unlike opinion 
and satisfaction surveys, these instruments usually have a right answer and the student is 
judged on his/her ability to select the best (right) answer. Some instruments contain a mix 
of discipline topics, but it is more common for tests to be designed to measure one specific 
knowledge domain. The use of these instruments may vary depending on the timing of 
the test. For example: 

• Surveys given during new-student orientations are often designed to place 
students in the appropriate level of college courses based on knowledge at 
the point of admissions. 

• Surveys given in the sophomore / junior year may serve as formative 
evaluation of progress or be "gateways" to a major. 

• Surveys given in the senior year may serve as summative evaluation or as a 
posttest of institutional effectiveness. 

In addition to knowledge testing, students may also be asked to self-report their gain 
in academic skills. 
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Instruments Sorted by Type 

A. Pre-Enrollment /Baseline Data 

• CSXQ - College Student Expectations Questionnaire (Kuh, Indiana) 

• CSI - College Student Inventory Form A & Form B (Noel-Levitz) 

• CIRP - The Freshman Survey (Astin, UCFA) 

• Entering Student Survey (ACT) 

• Student Needs Assessment Questionnaire (ACT) 

• Survey of Current Activities and Plans (ACT) 

• Survey of Postsecondary Plans (high school version) - (ACT) 

B. Surveys of Outcomes and Experiences in the First College Year 

• College Student Report - NSSE - National Survey of Student Engagement (Kuh, 
Indiana) 

* also a survey of seniors 

• CCSSE - Community College Survey of Student Engagement (McClenney, University 
of Texas) 

• Your First College Year (Sax, UCLA) - posttest to CIRP 

C. General Surveys (Behavior, Attitudes, Study Skills, Satisfaction, & Experiences) 

• College Outcomes Survey (ACT) 

• CSEQ - College Student Experiences Questionnaire (Kuh, Indiana) 

• College Student Needs Assessment Survey - (ACT) 

• CSS - College Student Survey (Astin, UCLA) 

• CSFI - College Success Factors Index (Ombudsman Press) 

• CCSEQ - Community College Student Experiences Questionnaire (Murrell, Memphis) 

• Core Alcohol and Drug Survey (Core Institute) 

• Faces of the Future (ACT / American Association of Community Colleges) 

• Institutional Priorities Survey — four-year and community/ junior college versions 
(Noel-Levitz) 

• LASSI - Learning and Study Strategies Inventory (Weinstein) 

• PEEK - Perceptions, Expectations, Emotions, and Knowledge about Campus (Wein- 
stein) 

• RSVP - Student Retention Survey — four-year and two-year versions (Harris Interna- 
tional) 

• SACQ - Student Adaptation to College Questionnaire (Western Psychological Ser- 
vices) 

• Student Development Task and Lifestyle Assessment (Student Development Associ- 
ates) 

• Student Opinion Survey - (ACT) 

• SSI - Student Satisfaction Inventory — four-year and community / junior college ver- 
sions (Noel-Levitz) 

• Survey of Student Opinions (ACT) 

• Study of College Health Behaviors (Harvard School of Public Health) 

• TheHealthSurvey (Outside The Classroom) 

D. Surveys of Specific Services / Units / Programs 

• ACUHO-I Resident Halls (EBI & Association of College and University Housing Of- 
ficers - International) 
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• College Student Unions (EBI & Association of College Unions - International) 

• Developmental Academic Advising (DAI) 

• Financial Aid Services - (ACT) 

• FYI - First Year Initiative Benchmarking (EBI) 

• FCEQ36 - Eearning Community Effectiveness Survey (Indiana) 

• Survey of Academic Advising - (ACT) 

E. Surveys of Specific Populations 

• Adult Learner Needs Assessment Survey (ACT) 

• Adult Student Priorities Survey (Noel-Levitz) 

• Fraternity Survey and Sorority Survey (EBI) 

• Withdrawing /Nonreturning Student Survey (short & long forms) (ACT) 

F. Placement and Academic Knowledge Surveys / Tests 

• Academic Profile (long & short forms) (ETS) 

• Accuplacer & Companion (College Board) 

• ASSET (ACT) 

• California Critical Thinking Dispositions Inventory (California Academic Press) 

• California Critical Thinking Skills Test (California Academic Press) 

• College BASE (Missouri) 

• College Placement Test (College Board, CPT) 

• Collegiate Assessment of Academic Proficiency (CAAP) 

• COMPASS /ESL (ACT) 

• Cornell Critical Thinking Test (Critical Thinking Press & Software) 

• Watson-Glaser Critical Thinking Appraisal (Psychological Corporation) 
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Instruments Sorted Alphabetically 

Academic Profile (long & short forms) Section F 

Accuplacer & Companion Section F 

ACUHOI Resident Hall Benchmarking Section D 

Adult Learner Needs Assessment Survey Section E 

Adult Student Priorities Survey Section E 

ASSET Section F 

California Critical Thinking Dispositions Inventory Section F 

California Critical Thinking Skills Test Section F 

College BASE Section F 

College Outcomes Survey Section C 

College Placement Test Section F 

CSXQ - College Student Expectations Questionnaire Section A 

CSEQ - College Student Experiences Questionnaire Section C 

College Student Needs Assessment Survey Section C 

College Student Report (National Survey of Student Engagement) Section B 

CSS - College Student Survey Section C 

CSI - College Student Inventory Form A & Form B Section A 

College Student Unions Section D 

CSFI - College Success Factors Index Section C 

Collegiate Assessment of Academic Proficiency Section F 

CCSEQ - Community College Student Experiences Questionnaire Section C 

CCSSE - Community College Survey of Student Engagement Section B 

COMPASS /ESL Section F 

CIRP - Cooperative Institutional Research Program The Freshman Survey Section A 

Core Alcohol and Drug Survey Section C 

Cornell Critical Thinking Test Section F 

Developmental Academic Advising Section D 

Entering Student Survey Section A 

Faces of the Future Section C 

Financial Aid Services Section D 

FYI - First Year Initiative Benchmarking Section D 

Fraternity Survey Section E 

Freshman Survey (see CIRP) Section A 

Institutional Priorities Survey Section C 

LASSI - Learning and Study Strategies Inventory Section C 

LCEQ36 - Learning Community Effectiveness Survey Section D 

National Survey of Student Engagement (see College Student Report) Section B 
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PEEK - Perceptions, Expectations, Emotions & Knowledge about College Section C 

RSVP - Student Retention Survey Section C 

Sorority Survey Section E 

S ACQ - Student Adaptation to College Questionnaire Section C 

Student Development Task and Lifestyle Assessment Section C 

Student Needs Assessment Questionnaire Section A 

Student Opinion Survey Section C 

SSI - Student Satisfaction Inventory Section C 

Study of College Health Behaviors Section C 

Survey of Academic Advising Section D 

Survey of Current Activities and Plans Section A 

Survey of Postsecondary Plans (high school version) Section A 

Survey of Student Opinions Section C 

TheHealthSurvey Section C 

Withdrawing /Nonreturning Student Survey Section E 

Watson-Glaser Critical Thinking Appraisal Section F 

YFCY - Your First College Year Section B 
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Section A: 

Pre-Enrollment/Baseline Data 

CSXQ - College Student Expectations Questionnaire 
Sponsoring Organization: Center for Postsecondary Research, Indiana University Bloom- 
ington, 1900 East 10 th Street, Eigenmann Hall 419, Bloomington, IN 47406-7512. (812) 856- 
5825. 

Contact: George Kuh 

Cost: $0.75 per test booklet, $125.00 set-up fee per administration, plus $1.50 per returned 
survey 

Key Features: Survey of student experiences in college. Includes student self-reported 
gains. Features curricular and co-curricular activities. 

Web Address: www.indiana.edu/ ~cseq/ csxq_generalinfo.htm 

Other Notes: Can be matched to CSEQ to compare expectations and experiences 

College Student Inventory 

Sponsoring Organization: USAGroup, Noel-Levitz, 2101 ACT Circle, Iowa City, I A 52245- 
9581. (800) 876-1117. 

Principal Investigator: Michael L. Stratil, Noel-Levitz Retention Management System 
Cost: $2.00 per reusable booklets each. $4.70 to $6.20 per answer sheet (includes processing 
and reports), based on number used. $7.25 each for online version, plus $60.00 set-up fee. 
Key Features: Intended to be part of an early warning program. Based on self-report of 
academic readiness and attitudes. Detailed background questions. 

Web Address: www.noellevitz.com/ solutions/ retention/ rms_csi/ index. asp 

Other Notes: Is marketed as part of a retention management system that includes guides 

for advisors and special reports. 

CIRP - The Freshman Survey 

Sponsoring Organization: Higher Education Research Institute, Cooperative Research 
Program at the University of California - Los Angeles. Moore Hall, Box 951421, University 
of California - Los Angeles, Los Angeles, CA 90095-1521. (310) 825-1925. 

Contact: Linda J. Sax 

Cost: $400.00, plus $1.50 per survey returned (fall 2003). $1.00 per survey returned after 
first 1,000 surveys. 

Key Features: Designed to survey entering college freshmen. Norm groups built from 
schools with 85% participation rates. Started in 1966 - the oldest data set on college fresh- 
men. Longitudinal data is a strength. Institutions administer the survey on campus and 
return them for processing. 

Web Address: www.gseis.ucla.edu/heri/freshman.htm 

Other Notes: May be combined with data from the Your First College Year (YFCY) survey 
to form a pre- and post-test of the first college year 

Entering Student Survey 

Sponsoring Organization: ACT, ESS Customer Services, ACT, P.O. Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893. 

Cost: $16.00 per 25, $65.00 institutional fee, $0.95 per form scanned 

Key Features: Measures reasons for attending and selecting current college, extracurricular 
activities desired, and perceptions of selected factors that influenced college selection 
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Web Address: www.act.org/ess/ postsec.html 

Other Notes: Space for 30 institutionally defined questions. National data available for 
comparison with local results. 

Student Needs Assessment Questionnaire 

Sponsoring Organization: ACT, ESS Customer Services, ACT, P.O. Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893. 

Cost: $16.00 per 25, $65.00 institutional fee, $0.95 per form scanned 
Key Features: Measures life goals, growth and development issues, and high school aca- 
demic needs 

Web Address: www.act.org/ ess/ second.html 

Other Notes: This instrument is for use with high school students. A companion survey is 
designed for college students. Could be used in a K-12/ College collaboration project. 

Survey of Current Activities and Plans 

Sponsoring Organization: ACT, ESS Customer Services, ACT, P.O. Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893. (Select "ESS" on automatic answering system.) 

Cost: $16.00 per 25, $65.00 set-up fee, $0.95 per form scanned 

Key Features: Measures impressions of the institution, sources of information, college 
selection plans, and educational objectives 
Web Address: www.act.org/ess/ postsec.html 

Other Notes: The stated goal of this survey is to collect information on students who were 
admitted but did not enroll. Could also be used with pre-admission populations or with 
first-year students. 



Survey of Postsecondary Plans 

Sponsoring Organization: ACT, ESS Customer Services, ACT, PO Box 1008, Iowa City, IA 
52243-1008. (319) 337-1893 - select "ESS" on automatic answering system. 

Cost: $16.00 per 25, $65.00 set-up fee, $0.95 per form scanned 

Key Features: Measures high school student plans for college or work, reasons for continu- 
ing education, impressions of selected college, reasons for selecting a college, and sources 
of information about selected college 
Web Address: www.act.org/ess/ postsec.html 

Other Notes: Aimed at pre-enrollment timing. Could be used as a pre-test or as part of a 
K-12 college initiative. 



Section B: 

Surveys of Outcomes and Experiences in the First College Year 



The College Student Report - National Survey of Student Engagement 
Sponsoring Organization: Center for Postsecondary Research, Indiana University Bloom- 
ington, 1900 East Tenth Street, Eigenmann Hall Suite 419, Bloomington, IN 47406-7512. 
(812) 856-5824. 

Contact: George Kuh 

Cost: $300.00 institutional participation fee, plus a survey fee based on size of institution: 
<4,000 = $3,150; 4,000 - 15,000 = $4,900; >15,000 = $7,000. Fee includes a set sample size; 
however, additional students may be surveyed for $1.50 (web), $1.50 paper survey with 
local collection, or $7.00 (mailed paper survey). 
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Key Features: Sample selection by NSSE. Mail and/ or web versions available. 

Web Address: www.indiana.edu/ ~nsse/ 

Other Notes: Similar to the CSEQ in content, but a structured sampling method assures 
comparable scores across institutions. Comparative data with similar institutions. Sam- 
ple includes first-year students and seniors. First national report released November 13, 
2000. 



Community College Survey of Student Engagement 
Sponsoring Organization: Community College Leadership Program, The University of 
Texas at Austin, George I. Sanchez Building, Suite 350A Austin, Texas 78712-1293. (512) 
471-6807. 

Contact: Kay M. McClenney 

Cost: Based on institutional size, 1,500 - 4,499 = $4,500.00, 4,500 - 7999 = $6,000.00, 8,000 
- 14,999 = $7,500. 15,000 or more = $9,000. Fees include set sample sizes, but additional 
students may be surveyed. The cost of additional surveys varies depending on type of 
survey administration. 

Key Features: Measures learning-centered indicators of quality for community colleges. 
Serves as a benchmarking instrument, establishing national norms for educational practice 
and performance by community and technical colleges; a diagnostic tool, identifying areas 
in which a college can enhance students' educational experiences; and a monitoring device, 
documenting and improving institutional effectiveness over time. 

Web Address: www.ccsse.org 
E-mail: info@ccsse.org 

Other Notes: First Open administration was Spring 2001. 

Your First College Year 

Sponsoring Organization: The Policy Center for the First Year of College and Higher Educa- 
tion Research Institute at UCLA, Moore Hall, Box 951521, University of California Los Angeles, 
Los Angeles, CA 90095-1521. (310) 825-1925. Funding by The Pew Charitable Trusts. 
Contact: Linda J. Sax 

Cost: 2003 participation fee $450.00, plus $2.00 for each returned survey 

Key Features:Matches with the CIRP Freshman Survey to create a measure of gain/ change 

during the first college year. Local administration with paper and/ or web options. 

Web Address: www.gseis.ucla.edu/heri/yfcy 

Other Notes: Spring 2002 was the first full administration, following two years as a pilot 
instrument. 



Section C 

General Surveys: Student Behavior, Attitudes, and Experiences 

College Outcomes Survey 

Sponsoring Organization: ACT, ESS Customer Services, ACT, P.O. Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893 - select "ESS" on automatic answering system. 

Cost: $16.00 per 25, $65.00 set-up fee, $0.95 per form scanned 

Key Features: 

Student self-report of gains, personal growth, social growth, cognitive achievement, basic 
skills knowledge acquisition, time allocations to work and study, perceived contributions 
of college to gains, satisfaction with college. 
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Web Address: www.act.org/ess/ postsec.html 
E-Mail: outcomes@act.org 

Other Notes: Space for 30 additional questions. Small space for one open-ended ques- 
tion. National data available for comparison with local results. Often used in community 
colleges as an end-of-second-year exit exam. 

CSEQ - College Student Experiences Questionnaire 
Sponsoring Organization: Center for Postsecondary Research, Indiana University 1900 
East 10th Street, Eigenmann Hall 419, Bloomington, IN 47406-7512. (812) 856-5825. 
Contact: George Kuh 

Cost: $0.75 per test booklet, $125.00 set-up fee, plus $1.50 per returned survey 

Key Features: Survey of student experiences in college. Based on the "quality of effort" 

work of Robert Pace. Includes self-reported gains in academic skills. 

Web Address: www.indiana.edu/ ~cseq/ 

Other Notes: Can be matched to CSXQ to determine the difference between expecta- 
tions and experiences. Web version also available. See also the CCSEQ for community 
colleges. 



College Student Needs Assessment Survey 
Sponsoring Organization: ACT, ESS Customer Services, ACT, P.O. Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893. 

Cost: $16.00 per 25, $65.00 institutional fee, $0.95 per form scanned 

Key Features: Goals for college, life goals, career development, educational planning, 

life and intellectual skills 

Web Address: www.act.org/ ess /postsec.html 

Other Notes: Room for 30 additional questions and space for open-ended response to the 
question: "If you wish to make any comments or suggestions concerning this college, 
write them on the lines below." 

CSS - College Student Survey 

Sponsoring Organization: Higher Education Research Institute, Cooperative Research 
Program, Moore Hall, Box 951421, University of California - Los Angeles, Los Angeles, 
CA 90095-1521. (310)825-1925. 

Contact: Alexander Astin 

Cost: $450.00, plus $2.00 per survey returned 

Key features: Can be used as a senior survey or general student survey. Data can be 
linked to Freshman Survey (CIRP) student responses. Institutions administer the survey 
to students and return them to HERI for processing. 

Web Address: www.gseis.ucla.edu/heri/ css.htm 

College Success Factors Index 

Sponsoring Organization: Ombudsman Press, Inc., 6938 Merrywood Court, Granite 
Bay, CA 95746. (916) 791-9290. 

Contact: Ed Hallberg 

Cost: call 800-400-7609 for quote 

Key Features: Measures control / responsibility, competition, task precision, expectations, 
wellness, time management, college involvement, and family / other involvement. Can 
be used as early warning or for self-assessment. Self-scoring. Web form available from 
Wadsworth Publishing 



Typology of Instruments 



185 



Web Address: www.schoolsuccesscentral.com or www.csii-wadsworth.com 

Community College Student Experiences Questionnaire 
Sponsoring Organization: Center for the Study of Higher Education, The University of 
Memphis, Memphis, TN 38152. (901) 678-2775. 

Contact: Patricia H. Murrell 

Cost: Test booklet / answer form $0.75, scoring per form $1.50, diskette and printout 
$125.00. 

Key Features: Measures student quality of effort. Built from the work of Robert Pace. This 
is the community college version of the CSEQ. 

E-Mail: ccseqlib@Memphis.edu 

Web Address: www.people.Memphis.edu/ ~coe_cshe/ ccseq_main.htm 

Core Alcohol and Drug Survey 

Sponsoring Organization: Core Institute, Center for Alcohol and Other Drug Studies, 
Southern Illinois University at Carbondale, Carbondale, Illinois 62901 (618) 453-4366. 
Cost: $0.20 per survey form, plus $0.04 shipping per form. Tabulation costs vary, starting 
at $50.00 for a standard cross-tabulation report. 

Key Features: Questions focus on student perceptions and opinions about alcohol and other 
drugs, and their use of alcohol and drugs. The instrument can be used as a pre / posttest to 
evaluate the effectiveness of prevention programs. National norms are available. 

Web Address: www.siu.edu/ departments/ coreinst/ public_html/ index.html 

Faces of the Future 

Sponsoring Organization: ACT and American Association of Community Colleges 
Contact: Kent A. Phillippe 

Cost: $16.00 per 25, $65.00 institutional fee, $0.95 per form scanned 

Key Features: Especially designed for community colleges and technical schools. Measures 

reasons for attending this college, satisfaction with the college, impression of the campus 

climate, and areas of growth while at the college 

Web Address: www.act.org/ ess/ postsec.html 

Institutional Priorities Survey (Two versions) 

Sponsoring Organization: Noel-Levitz, 2101 ACT Circle, Iowa City, IA 52245-9581. (800) 
867-1117. 

Cost: Set-up fee $150.00, cost per form varies based on quantity: $1.50 each for 2,500 or 
more; $1.75 each for 1,000-2,499; $1.95 each for 100-999; $2.95 each for 1 to 99 
Key Features: Measures "level of importance" compared to "level of satisfaction." Avail- 
able in a four-year version and a community, junior, and technical college version 
Web Address: www.noellevitz.com/ solutions/ retention/ satisfaction/ ips/index.asp 

Fearning And Study Strategies Inventory (FASSI) 

Sponsoring Organization: H&H Publishing Company, Inc. 1231 Kapp Drive, 
Clearwater, Florida 33765-2116. (800) 366-4079. 

Contact: Claire Weinstein 

Cost: $2.75 each for 100 or more, $3.20 each for 1 to 99 

Key Features: Gathers information about student study patterns. Can be used as 
a diagnostic measure, counseling tool, pre / posttest of an intervention, or part of a 
retention management program. Self scoring. Electronic version available. 
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Web Address: www.hhpublishing.com/_assessments/LASSI/ index.html 

PEEK 

Sponsoring Organization: H&H Publishing Company, Inc., 1231 Kapp Drive, Clearwater, 
Florida 33765-2116. (800) 366-4079. 

Contact: Claire Weinstein 

Cost: $1.25 each for 100 or less if institution scans and scores, $1.75 each if H&H scans and 
scores. Discount for 500 or more surveys. 

Key Features: Thirty questions long. Available in web-based and disk-based forms. 
Results allow institutions to target intervention programs to high-risk students. Measures 
academic, personal, and social dimensions of the college experience. 

Web Address: www.hhpublishing.com/_assessments/PEEK/index.html 

Student Adaptation to College Questionnaire 
Sponsoring Organization: Western Psychological Services, 12031 Wilshire Blvd., Los 
Angeles, CA 90025-1251. (800) 648-8857. 

Cost: pack of 25 (hand scored) - $39.50. Mail-in answer sheets $13.00 each, 25-user disk 
version - $180.00 

Key Features: Measures academic adjustment, personal-emotional adjustment, social 
adjustment, and attachment to the institution. Can be used as an early warning test. 
Research shows that the results encourage those in need to seek counseling help. 

Web Address: https: / / www-secure.earthlink.net / www.wpspublish.com / Inetpub4 / catalog / 
W-228.htm 



Student Developmental Task and Fifestyle Assessment 
Sponsoring Organization: Student Development Assessment, Inc., PMB 500, 2351 College 
Station Road, Athens, GA 30605-3664. 

Cost: Site license options $1,000 for 12 months. Special graduate student research pricing. 
Key Features: Measures three vectors of personal development (a) establishing and 
clarifying purpose, (b) developing autonomy, and (c) establishing mature interpersonal 
relationships. Each task has subtasks that are scored independently. Can be used as a pre/ 
posttest to measure change. 

Web Address: www.geocities.com/ studevassoc/ SDTLALINK.htm 

Student Opinion Survey (Four-year & two-year versions) 

Sponsoring Organization: ACT, ESS Customer Services, ACT, PO. Box 1008, Iowa City, 

IA 52243-1008. (319) 337-1893 - select "ESS" on automatic answering system. 

Cost: $16.00 per 25, $65.00 set-up fee, $0.95 per form scanned 

Key Features: Pairs "usage" of college services with "satisfaction" for each satisfaction 
score. Collects student opinion about academics, admissions, rules and policies, facilities, 
registration, and general college experiences. 

Web Address: www.act.org/ ess/ postsec.html 

Other Notes: Space for 30 additional questions. One open-ended question: "If you wish to 
make any comments or suggestions concerning this college, please write them on the lines 
below." National data available for comparison with local results. 

Student Satisfaction Inventory 

Sponsoring Organization: USAGroup, Noel-Levitz, 2101 ACT Circle, Iowa City, IA 52245- 
9581. (800) 876-1117. 
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Cost: Processing/ set-up fee $50.00, plus $1.50 each for 2,500 or more; $1.75 each for 1,000 
to 2,499; $1.95 each for 100 to 999; and $2.25 each for online administration. 

Key Features: Designed for annual use. Asks students to rate the importance of services 
and their satisfaction with those services. Creates a scale of "meets, exceeds, falls below 
expectations" For use with students who have experience on the campus (end of year 
first-year students, sophomores, juniors, and seniors) 

Web Address: www.noellevitz.com/ solutions/ retention/ satisfaction/ ssi / index. asp 

Study of College Health Behaviors 

Sponsoring Organization: Harvard School of Public Health, 677 Huntington Ave, 
Boston, MA 02115. (301) 652-1558. 

Cost: $6,000.00 for basic survey package, including surveys for 750 students. 

Key Features: Related to the College Alcohol Study, but looks at a broader set of 
behaviors including: alcohol use, tobacco use, illicit drug use, stress, mental health, 
sleep, sexual health, physical activity and weight control, motor vehicle safety, and use 
of health services. Administered via the web. 

Web Address: www.hsph.harvard.edu/ cas/ chb/ 

Notes: This is a new survey for 2004. 

Survey of Student Opinions 

Sponsoring Organization: ACT, ESS Customer Services, ACT, PO Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893 - select "ESS" on automatic answering system. 

Cost: $16.00 per 25, $65.00 set-up fee, $0.95 per form scanned 
Key Features: Focuses on college services and college environments. Compares 
"importance" and "satisfaction." for each item. More demographics than other ESS 
forms. This is the newest survey in the series (copyright 2002). For use with students 
who have had experience with a campus, because they are asked to rate experiences 
with campus units and services. 

Web Address: www.act.org/ess/ postsec.html 

Other Notes: Space for 30 additional questions. One open-ended question: "If you wish 
to make any comments or suggestions concerning this college, please write them on the 
lines below." 



TheHealthSurvey 

Sponsoring Organization: Outside The Classroom, 385C Elliot Street, Newton, MA 
02464. (617) 641-2001. 

Cost: $750.00 participation fee, plus $0.35 to $0.40 per survey form. 

Key Features: Survey contains items about alcohol, tobacco, and other drugs; sexual 
behavior; sexual assault; eating patterns and disorders; and mental health. Outside The 
Classroom will survey a random sample of all students, or institutions may survey only 
special populations such as only first-year students. 

Web Address: www.outsidetheclassroom.com 

Other Notes: Up to 10 custom questions can be added to the survey. 
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Section D: 

Surveys of Specific Services/Units/Programs 

ACUHO-I/EBI Resident Satisfaction Survey 
Sponsoring Organization: Association of College & University Housing Officers 
International and Educational Benchmarking, Inc. 1630 W. Elfindale, Springfield, MO 
65807. (417) 831-1810. 

Contact: Joe Pica and Glen Detrick 

Cost: $1,050 for up to 500 surveys. $0.25 per additional survey 

Key Features: Measures resident student satisfaction, assessment of resident advisor/ 
assistant performance, facilities, dining services, alcohol use, future plans 
Web Address: www.webebi.com/Housing/Resident/Resident.htm 
E-Mail: info@webebi.com 

Other Notes: Has space for 10 institutionally defined questions. Institution sets 
housing code to identify building and floor /wing. 

College Student Unions 

Sponsoring Organization: Association of College Unions - International and 
Educational Benchmarking, Inc. 1630 W. Elfindale, Springfield, MO 65807. (417) 831- 
1810 

Contact: Glen Detrick and Joe Pica 
Cost: $995.00 for 500 surveys 

Key Features: Measures satisfaction with college union facilities and programs, 
student use of the college union, food services in the college union, and assesses 
bookstore. 

Web Address: www.webebi.com/Unions/ CollegeUnionStudentCenter/CollegeUnio 

nStudentCenterDetails.htm 

E-mail: info@webebi.com 

Other Notes: Space for 10 institutionally defined questions. 

Developmental Academic Advising Inventory 
Sponsoring Organization: Developmental Academic Advising Inventory, Inc. P.O. 
Box 1946, Paradise, CA 95967. (530) 872-0511. 

Cost: Based on quantity and usage agreement - $1.00 and up. Special pricing for 
research use. 

Key Features: Web and paper forms. Includes scales for intellectual, life planning, 
social, physical, emotional, sexual, cultural, spiritual, and political. Can be used in 
advising settings or other student development programs. 

Web Address: daiassess.com/introduction.htm 
E-mail: gdickson@sunset.net 



Financial Aid Services 

Sponsoring Organization: ACT, ESS Customer Services, ACT, P.O .Box 1008, Iowa 
City, IA 52243-1008. (319) 337-1893 - select "ESS" on automatic answering system. 
Cost: $16.00 per 25, $65.00 set-up fee, $0.95 per form scanned 

Key Features: Is a general assessment of financial aid procedures, staff, publications, 
and results. Collections demographic data and has space for up to 30 additional items 
Web Address: http: / / www.act.org/ ess/ postsec.html 
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First Year Initiative (Benchmarking Survey) 

Sponsoring Organization: Policy Center on the First Year of College and Educational 
Benchmarking, Inc. 1630 W. Elfindale, Springfield, MO 65807. (417) 831-1810. 

Contact: Glen Detrick and Joe Pica 

Key Features: Specially designed to assess learning outcomes from first-year seminars. 
Uses a benchmarking process which provides comparative data. 

Cost: $695.00 for 250 surveys, $.35 per additional survey 

Web Address: http: / / www.webebi.com/University/FYI/Index.htm 

Other Notes: The first pilot of the instrument was conducted during Spring 2001. 

LCEQ36 -Learning Communities Experience Questionnaire 
Sponsoring Organization: Indiana State University, Office of Strategic Planning and 
Institutional Research and Effectiveness, 200 North Seventh Street, Terre Haute, IN 
47809-9989. (812) 237-7778. 

Contact: Kevin J.G. Snider and Ann M. Venable 
Cost: Free with permission 

Key Features: Has both pretest and posttest versions. Assesses learning communities 
and first-year seminars. Scales include: student-student collaboration, faculty-student 
collaboration, academic involvement, perspectivism, cooperative learning, linking 
academic and life experience, and interdisciplinary learning. Covers behavioral, 
cognitive, and motivational learning domains. 

Web Address: web.indstate.edu/ oirt/lce36/1999Study/ 

Other Notes: See web site for a PowerPoint presentation on the first year's study. 

Survey of Academic Advising 

Sponsoring Organization: ACT, ESS Customer Services, ACT, PO Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893 - select "ESS" on automatic answering system. 

Cost: $16.00 per 25, $65.00 set-up fee, $0.95 per form scanned 

Key Features: Compares advising services received and satisfaction with services 

Measures impressions of academic advisor. Can be used with "professional" advisors 

and faculty advisors. Room for 30 institutionally defined questions. Contains some 

student behavioral questions about use of advising services 

Web Address: www.act.org/ess/ postsec.html 

Other Notes: Contains an open-ended question: "If you wish to make any comments 
or suggestions concerning this college or its advising program, please use the lines 
provided below." National data available for comparison with local results. 

Section E: 

Surveys of Specific Populations 

Adult Learner Needs Assessment Survey 

Sponsoring Organization: ACT, ESS Customer Services, ACT, PO Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893 

Cost: $16.00 per 25, $65.00 institutional fee, $0.95 per form scanned 
Key Features: Aimed at point of entry, but could be used for enrolled students. Asks 
about work, family support, and college financing. Major divisions include: life skill 
development, career development, educational planning, and relationships with others 
Web Address: www.act.org/ess/ postsec.html 
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Adult Student Priorities Survey 

Sponsoring Organization: USAGroup, Noel-Levitz, 2101 ACT Circle, Iowa City, IA 
52245-9581. (800) 876-1117. 

Key Features: Designed especially for adult students. Compares opinion of 
"importance" compared with "satisfaction." For students 25 years and older. 

Web Address: www.noellevitz.com/ solutions/ retention/ satisfaction/ asps/ index.asp 

Fraternity & Sorority Survey 

Sponsoring Organization: Educational Benchmarking, Inc. 1630 W. Elfindale, 
Springfield, MO 65807. (417) 831-1810. 

Contact: Glen Detrick and Joe Pica 

Cost: $1.00 per student for up to 3,500 students, minimum of 2,000. $0.50 per student for 
next 3,501 - 8,500 students, $0.35 per student for next 8,500 students 
Key Features: Measures Greek life, Greek advantages, self-reported gains from 
membership, fraternity house issues, national/ international services, chapter 
programming, food services, alcohol use. 

Web Address: www.webebi.com/ AFA_Fraternity_Sorority / National_index.htm 

Withdrawing/Nonreturning Student Survey (long form) 

Sponsoring Organization: ACT, ESS Customer Services, ACT, P.O. Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893 - select "ESS" on automatic answering system. 

Cost: $16.00 per 25, $65.00 set-up fee, $0.95 per form scanned 
Key Features: Major and minor reasons for leaving college: personal, academic, 
institutional, financial, and employment-related. Also collects future educational plans, 
satisfaction with college services and college characteristics. 

Web Address: www.act.org/ ess/ postsec.html 

Other Notes: Room for 30 institutionally defined questions. Open-ended question: "If 
you wish to make any comments or suggestions concerning the college, please write 
them on the lines below." 

Withdrawing/Nonreturning Student Survey (short form) 

Sponsoring Organization: ACT, ESS Customer Services, ACT, P.O. Box 1008, Iowa City, 
IA 52243-1008. (319) 337-1893 

Cost: $11.50 per 25, $65.00 institutional fee, $0.95 price per form scanned 
Key Features: Major and minor reasons for leaving college: personal, academic, 
institutional, financial, and employment-related. Also collects future educational plans, 
satisfaction with college services and college characteristics. 

Web Address: www.act.org/ ess/ postsec.html 

Other Notes: Room for 20 institutionally defined questions. 

Section F: 

Placement and Academic Knowledge Surveys/Tests 

Academic Profile (standard form) 

Sponsoring Organization: The College Board & Educational Testing Service (ETS) 55-L 
Educational Testing Service, Princeton, NJ 08541. (609) 921-9000. (800) 745-0269. 

Cost: $375.00 annual institutional fee, plus min. order 100. quantities of 25, $15.50 each, 
volume pricing on individual bases. 
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Optional Essay - $3.50 no additional charge if scored by local institution using ETS 
manual. $14.00 if scored by ETS. 

Key Features: Takes two hours to administer 108 questions. Measures reading, 
writing, mathematics, and critical thinking. Measurements in the context of the 
humanities, social science, and natural sciences — not recall of specific information. 
Seven norm referenced scores (i.e., reading, writing, mathematics, critical thinking, 
humanities, social science, and natural science). Norms for institution and national 
users. Criterion-referenced in skill areas. Three levels for each skill (i.e., writing, 
reading, and mathematics). Both individual and group scores are reported. 

Web Address: www.ets.org/hea/ acpro/ benefits.html 
Other Notes: See also the abbreviated form. 

Academic Profile (abbreviated form) 

Sponsoring Organization: The College Board & Educational Testing Service (ETS) 55-L 
Educational Testing Service, Princeton, NJ 08541. (609) 921-9000. (800) 745-0269. 

Cost: $375.00 annual institutional fee. 100 min. order, packages of 25, $13.50 each 
Key Features: Takes about 40 minutes to administer each unit of 36 questions. 
Measures reading, writing, mathematics, and critical thinking. Measurements in the 
context of the humanities, social science, and natural sciences — not recall of specific 
information. Seven norm referenced scores (i.e., reading, writing, mathematics, 
critical thinking, humanities, social science, and natural science). Norms for 
institution and national users. Criterion-referenced in skill areas. Three levels for 
each skill (i.e., writing, reading, and mathematics). The short form only produces 
group-level scores, no individual sub-scores. 

Web Address: www.ets.org/hea/ acpro /benefits.html 

Other Notes: National comparative data available by Carnegie classification, student 
class, and self-selected institutional groups. 

Accuplacer™ & Companion® 

Sponsoring Organization: The College Board, 45 Columbus Avenue, New York, NY 
10023. (800) 486-8497. 

Cost: $0.90 to $1.10 per test component. Writing essay is $3.60 to $4.40 per essay. 
Discounts for Texas, Florida, Connecticut, California, Colorado, North Carolina, 
Massachusetts, Maryland, and Minnesota 

Key Features: For placement, advisement, and guidance. Companion is the paper 
version and Accuplacer is the Internet version. Covers reading comprehension, 
sentence skills, elementary algebra, arithmetic. Essay is immediately graded using 
artificial intelligence and provides a holistic grade with sub-scores for focus, 
development, organization, and sentence structure. The Internet version must be 
proctored. 

Web Address: www.collegeboard.com/ accuplacer /html/ accuplal.html 

ASSET 

Sponsoring Organization: 500 ACT Drive, PO Box 168, Iowa City, IA 52243. (319) 337- 
1376. (800) 498-6065. 

Cost: $36.75 per 25 reusable test booklets, plus $3.55 to $2.85 for student sets (answer 
& information - not reusable). Scoring includes: self scoring carbon answer set, PC 
scoring, and scan scoring. 

Key Features: Paper and pencil placement test and advising program used by more 
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than 500 community and technical colleges. Academic skill placement measures in 
writing, reading, and mathematics with immediate advising and placement services. 
Optional units on study skills and interests. 

Web Address: www.act.org/ catalog / asset.html 
Other Notes: Designed for group administrations. 

California Critical Thinking Dispositions Inventory 
Sponsoring Organization: California Academic Press. 217 La Cruz Avenue, Milbrae, CA 
94030. (650) 697-5628. 

Cost: Varies depending on number of instruments and whether scoring is included. 25 
copy packages of CCTDI, $110; 100 copy packages with scoring, $485. 

Key Features: Designed to assess individual's inclinations toward or habits of mind 
considered to be contributing factors to critical thinking. These include truthseeking, 
openmindedness, analyticity, systematicity, critical thinking self-confidence, 
inquisitiveness, and cognitive maturity. 75-item, 6-point Likert scale, 20 minutes to 
administer. 

Web Address: http: / / www.insightassessment.com/ test-cctdi.html 

California Critical Thinking Skills Test 

Sponsoring Organization: California Academic Press. 217 La Cruz Avenue, Milbrae, CA 
94030. (650) 697-5628. 

Cost: Varies depending on number of instruments and whether scoring is included. 25 
copy packages of CCTDI, $100; 100 copy packages with scoring, $460. 

Key Features: CCTST is based on the American Philosophical Association 1990 Delphi 
report consensus definition of critical thinking. 34-item multiple-choice instrument, 
available in parallel Forms A and B. 

Web address: http:/ / www.insightassessment.com /test-cctdi.html 

College BASE 

Sponsoring Organization: Assessment Resource Center, University of Missouri. 2800 
Maguire Boulevard, Columbia, Missouri 65201. (800) 366-8232. 

Cost: Prices vary by number of subjects tested and size of institution. 

Key Features: Designed to assess general education/ core curriculum. Each of four 
multiple-choice tests may be used alone. Each takes 45 minutes. A writing prompt is also 
available and takes 40 minutes to administer. 

Web Address: arc.missouri.edu/ collegebase/ 

Other Notes: College Base is used as a state-wide test in Missouri and tied to 
performance funding. It is often used as a senior/ exit exam or a junior exam as the 
capstone of the general education experience. 

Collegiate Assessment of Academic Proficiency (CAAP) 

Sponsoring Organization: ACT Outcomes Assessment, PO Box 168, Iowa City, IA 
52243-0168. (319) 337-1053. 

Cost: (2001 fees) $330.00 annual participation fee. 1-500 students (each taking one 
test area) $11.00 each, (each taking 2-5 test areas) $17.00. 501 - 1,500 students (each 
taking one test area) $10.20 each. (each taking 2-5 test areas) $15.70. Writing Essay 
Test - $5.00 if scored locally, $11.00 if scored by ACT (for 2000-2001 academic year) 
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Key Features: Writing Skills Test (72-items, 40 minutes, multiple choice), Reading 
Test (36-items, 40 minutes, multiple choice). Science Reasoning Test (45 items, 40 
minutes, multiple choice). Mathematics Test (35 items, 40 minutes, multiple choice). 
Critical Thinking Test ( 32 items, 40 minutes, multiple choice). Writing Essay Test 
(Two 20-minute writing tasks). 

Web Address: http:/ / www.act.org/ caap/ index.html 
E-mail: outcomes@act.org 

Other Notes: National user norms available. Can be matched to ASSET and 
COMPASS to create a longitudinal measure of change. 

COMPASS/ESL 

Sponsoring Organization: ACT Customer Services, ACT, PO Box 1008, Iowa City, 
IA 52243-1008. (800) 645-1992. (319) 337-1054. 

Cost: Computer based: $450.00 per license, $1.05 to $1.30 per unit (example, one 
placement test + creating a student record = 1.40 unit). Institution must dedicate a 
PC for each license. 

Key Features: Placement and diagnostic tests: Writing, Reading, Mathematics, 
English as a Second Language. Computer adaptive testing. Collects background/ 
demographic data. Extensive reports for the institution and student. Immediate 
scoring. Non-timed tests. 

Web Address: www.act.org/ compass/ 

Other Notes: These tests were primarily developed for placement purposes 

Cornell Critical Thinking Test 

Sponsoring Organization: Critical Thinking Books and Software. P.O. Box 448, 
Pacific Grove, CA 93950. (800) 458-4849. 

Cost: Set of 10 tests, $17.99. Set of 10 answer sheets, $8.99. Administration manual, 
$8.99. 

Key Features: 50-item multiple-choice, taking 50 minutes. Level Z of the test is 
suitable for first-year college students. 

Web Address: www.criticalthinking.com/ series/ 055/ index_g.html 

Other Notes: Also available in a CD version; $39.99 for 10 students, $1.25 for each 

additional test taker. 



Watson-Glaser Critical Thinking Appraisal 
Sponsoring Organization: The Psychological Corporation. 19500 Bulverde Rd., San 
Antonio, TX 78259. (800) 872-1726 

Cost: Long forms (A & B): test booklets, 25 for $130; answer documents, 25 for $39. Short 
form: test booklets, $106; answer documents, 25 for $40. 

Key Features: Form S is composed of 40 multiple-choice items and takes 45 minutes to 
complete. Forms A and B (alternate versions) are 80 items each and take 60 minutes. 
Produces a single score based on five critical thinking skills: inference, recognition of 
assumptions, deduction, interpretation, and evaluation of arguments. 

Web Address: marketplace.psychcorp.com/PsychCorp.com/ Cultures/ en-US/ 
default.htm 

Other Notes: Promoted by the vendor primarily as a tool for employment applicant 
assessment. 
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Taylor W. Acee is a doctoral student in the Department of Edu- 
cational Psychology at the University of Texas at Austin. His 
area of study is learning, cognition, and instruction. His primary 
research and development interests include the development of 
goals, the transition from high school to college and from college 
to work, and applications designed to help students become more 
strategic learners. 



Catherine Anderson Catherine Anderson has served as academic advisor at the 
University of Mississippi's Academic Support Center for four 
years. In addition to her advising duties, she also coordinates 
the universities Absence-Based Intervention program. Anderson 
holds a BA in English from The University of Mississippi and 
an MA in English Literature from Western Kentucky University. 
She is currently pursuing a PhD in higher education, educational 
leadership at the University of Mississippi. 



Karen Webber Bauer Karen Webber Bauer is director of institutional research and as- 
sociate professor. Institute of Higher Education at The University 
of Georgia. Bauer is active in several professional associations, 
including the Association for Institutional Research and the 
Society for College and University Planning. She served as a 
contributor to several grants including two NSF grants on the 
evaluation of undergraduate research and a Pew Charitable Trust 
grant on problem-based learning. She has authored publications 
on institutional research, general education, higher education 
assessment, and the psychology of gender. 



Marcia Belcheir Marcia Belcheir is the coordinator of institutional assessment 
at Boise State University in Boise, Idaho. She holds a PhD in 
educational measurement and evaluation from the University 
of Florida. Her research interests include student learning and 
retention. 



Leonard Bliss Leonard Bliss is professor of educational research in the de- 
partment of educational and psychological studies at Florida 
International University in Miami. He received his PhD degree 
from Syracuse University in educational research and statistics. 
His current research interests involve study and self-regulatory 
behaviors of high school and university students. Bliss is a co- 
author of the Study Behavior Inventory, an instrument used by 
many colleges and universities in the United States and other 
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English-speaking countries. He has also produced a Spanish lan- 
guage edition of the instrument, the Inventario de Comportamiento de 
Estudio. He and his students are currently working on the validation 
of the Study Behavior Inventory - High School Version, which should 
be available in the spring 2004. 

Julie L. Bryant, senior director of retention products at Noel-Levitz, 
works directly with colleges and universities throughout North 
America in the area of satisfaction assessment. She partners with 
educators to determine relationships between perceptions of impor- 
tance and satisfaction of students, special populations, and campus 
personnel. Bryant identifies ways these data can inform retention 
planning and be shared with the campus community. She also over- 
sees the annual national reporting and trend analysis of these data. 
She is responsible for client service to more than 1,500 institutions 
using the Noel-Levitz family of satisfaction-priorities assessment 
instruments, including the Student Satisfaction Inventory™ and the 
Institutional Priorities Survey™. 

Lendol Calder is associate professor of history at Augustana College, 
Rock Island, IL, where he is chair of the history department. Before 
coming to Augustana, he taught at Colby-Sawyer College in New 
Hampshire, the University of Washington, Seattle, and the Univer- 
sity of Chicago, where he received his PhD in 1993. A scholar of the 
history of American consumerism, Calder 's 1999 book Financing the 
American Dream: A Cultural History of Consumer Credit has made him 
a popular authority in the national media on issues related to credit 
and debt. As a Carnegie Scholar at the Carnegie Academy for the 
Scholarship of Teaching and Learning, Calder's most recent work 
has examined the problem of "coverage" in introductory courses in 
an effort to help college and university instructors teach disciplinary 
thinking more effectively. 

Sarah-Eva Carlson is a 2002 graduate of Augustana College. A 
member of Phi Beta Kappa, Carlson studied history and worked as 
the lead research assistant on the think-aloud project described in 
this monograph. Carlson is now a graduate student at St. Andrews 
University, Scotland reading for an MLitt in international security 
studies. 

Yoonjung Cho is a doctoral student in the Department of Educational 
Psychology at the University of Texas at Austin. Her area of study 
is learning, cognition, and instruction. Yoonjung's primary research 
and development interests include the nature of goal orientations, 
the transition from high school to college and from college to work, 
and applications designed to help students become more strategic 
learners. She is also interested in cross-cultural studies and is an 
author of the Korean version of the LASSI-High School Version. 
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Marian Allen Claffey Marian Allen Claffey, MEd, is the special projects administrator for 
the office of the provost at Loyola University Chicago, where she is 
also completing a PhD in higher education. Claffey has more than 
20 years professional experience in academic and student affairs 
administration. Her appointments at Loyola have included assistant 
dean in the Graduate School of Arts and Sciences, project manager 
for accreditation and assessment in the office of the vice president 
for academic affairs, and associate dean in Mundelein College of 
Loyola, the university's adult, undergraduate college. She has 
presented both nationally and regionally on the topic of fairness 
in the academy. Claffey's research interests include organizational 
behavior and organizational justice in postsecondary settings. Her 
dissertation research will address college student perceptions of 
campus fairness. 

Stephanie B. Corliss Stephanie B. Corliss is a doctoral student in the Department of 
Educational Psychology at the University of Texas at Austin. She 
is also completing a master's degree in program evaluation. Her 
area of study is a combined program between learning, cognition, 
and instruction and instructional technology. Stephanie's primary 
research and development interests include instructional technology, 
cognitive theory, and ways to use technology effectively to enhance 
student learning. 

Marc Cutright Marc Cutright is an assistant professor in the department of coun- 
seling and higher education at Ohio University. He is a research 
fellow of The Policy Center on the Pirst Year of College, where he 
previously was on staff as a research associate. 



Garrick Davis Garrick Davis has taught first-year experience writing seminars at 
the California State University at Monterey Bay. He has also served 
as an editor of achievement tests at CTB / McGraw-Hill. His poetry 
and criticism have appeared in various publications. 



Teresa L. Flateby Teresa L. Flateby is director of evaluation and testing, at the Univer- 
sity of South Florida. Having served in that position for 13 years, 
she is currently directing the assessment of the Liberal Arts General 
Education program at USF. Serving as the university's representa- 
tive regarding institutional testing to the state, she has also directed 
the assessment of USF's learning communities, basic mathematics 
coursework, first-year experience course, and has supervised a staff 
of eight faculty and staff. Along with E. Metzger, she is the creator 
of the Cognitive Level and Quality of Writing Assessment. 



Shannon K. Gilmartin Shannon K. Gilmartin is a senior educational researcher at the Caltech 

Precollege Science Initiative (CAPSI) housed at the California Insti- 
tute of Technology. She received her PhD in education at the Univer- 
sity of California, Los Angeles, where she worked on the Your First 
College Year pilot studies. Her work explores peer relationships in 
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the first college year, survey methodology, and the science pipeline 
at middle- and high-school levels. 

Robert M. Gonyea is assistant director of the College Student Experi- 
ences Questionnaire Research Program and also serves as a research 
analyst for the National Survey of Student Engagement (NSSE). 
He has presented at several national and regional conferences on 
CSEQ, NSSE, and his own research, which focuses on assessment 
of institutional quality, student expectations about college, and the 
contribution of student affairs to student learning. Gonyea has previ- 
ously worked as a student affairs administrator at the University of 
San Francisco, the University of Wisconsin, and Northern Michigan 
University. Gonyea has a BS in psychology and a masters of labor 
and industrial relations from Michigan State University, and is a 
doctoral candidate in higher education and student affairs at Indiana 
University with a minor in educational inquiry. 

Edmond C. Hallberg is an emeritus professor of higher education 
and counseling at the California State University at Los Angeles. He 
has been a coordinator of academic advising, a dean of admissions, 
and a dean of students. His latest book. Making the Dean’s List, is a 
textbook for first-year seminars. 

Hayek is the senior associate director of the Indiana University Cen- 
ter for Postsecondary Research. He also manages the day-to-day 
operations of the National Survey of Student Engagement (NSSE). 
More than 425,000 students at 730 different colleges and universi- 
ties have participated in the program. Hayek has written on topics 
related to collegiate quality and institutional effectiveness and has 
given numerous presentations at national conferences. In addition, 
he has given invited talks on topics related to improving the col- 
lege experience at various staff and faculty retreats and workshops. 
Hayek has an economics degree from the University of Chicago and 
a doctorate in higher education from Indiana University. 

Associate professor, Elizabeth Hodges has been at Virginia Com- 
monwealth University since 1989. Her scholarly work lies in the 
field of composition studies. She also writes creative nonfiction and 
has published one book of connected essays. What the River Means. 
She is currently working on a second collection. 

J. Daniel House has been director of institutional research at North- 
ern Illinois University (NIU) since 1994. He has used CIRP data for 
numerous assessment projects. Those results have been presented 
at many conferences and published in the Journal of College Student 
Development, Research in Higher Education, and The Journal of Social 
Psychology. He serves on the Academic Planning Council and Uni- 
versity Assessment Panel at NIU. He has been an instructor at the 
Higher Education Research Institute (HERI) Summer Assessment 
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Workshops at UCLA since 1997. During those workshops he provides 
examples of how to incorporate CIRP data into a comprehensive 
campus assessment strategy. He earned a BS and MS in psychology 
at Illinois State University and a PhD in education at the University 
of Iowa. 

Peggy Pei-Hsuan Hsieh Peggy Pei-Hsuan Hsieh is completing her doctoral studies in the 

Department of Educational Psychology at the University of Texas at 
Austin. Her area of study is learning, cognition, and instruction. Her 
primary research and development interests include the transition 
from high school to college, strategies involved in learning a foreign 
language, and applications designed to help students become more 
strategic learners. 

Pamela A. Humphreys Since 1997, Pamela A. Humphreys has served as the senior College 

BASE coordinator at the Assessment Resource Center, University of 
Missouri-Columbia. She is a graduate of Truman State University 
and has completed post-graduate work at Mundelein College and the 
University of Missouri-Columbia. She comes to her current position 
after serving as education coordinator for national and regional medi- 
cal associations where she worked in the areas of continuing medical 
education, accreditation, and program development. She has more 
than 20 years experience in health care administration with appoint- 
ments in quality management initiatives and program evaluation. 

Roberta Jessen Roberta Jessen has been involved with the assessment of Portland 
State University's university studies program since its implementation 
in 1994. For several years she practiced participative observational 
research in the classroom and published with her colleagues Cheryl 
Ramette and Martha Balshem. She earned a BS in psychology at Port- 
land State University in 1987 and has nearly completed her graduate 
coursework in postsecondary education. Her interest has always been 
on the cognitive and affective process of student learning, especially 
as it relates to critical thinking and a caring community. She is now 
working as a professional academic advisor in the Academic and Sup- 
port Center at Portland State under the offices of student affairs. 

Angela L. Julie Angela L. Julie is a doctoral student in the Department of Educational 
Psychology at the University of Texas at Austin. Her area of study 
is learning, cognition, and instruction. She has a masters of educa- 
tion in teaching from the University of Texas at Arlington. She has 
extensive teaching experience in adult professional development, in 
high school mathematics, and in an undergraduate developmental 
education course in strategic learning. Angela's primary research 
and development interests include effective teacher education, pro- 
fessional certification, and the transfer of knowledge into classroom 
practice. 
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Cynthia A. King is a doctoral student in the Department of Edu- Cynthia A. King 
cational Psychology at the University of Texas at Austin. Her area 
of study is learning, cognition, and instruction, and her primary 
research and development interests include the transition from 
college to work and lifelong learning. She is currently complet- 
ing an internship evaluating the orientation program for first-year 
students in the Educational Psychology department at the Univer- 
sity of Texas at Austin. In addition, she is helping to develop and 
implement a new training program for assistant instructors for an 
undergraduate course called, "Individual Learning Skills." 

Ned Scott Laff, PhD, is coordinator for online programs and special Ned Scott Laff 

academic programs at Loyola University in Chicago. He has held 

numerous administrative positions, including assistant dean in 

the College of Liberal Arts at Plorida Atlantic University, assistant 

dean for first year students at St. Lawrence University, director for 

university advising at Weber State University, and associate dean 

for curriculum at Mundelein College of Loyola University Chicago. 

He has taught in both English and honors courses during each of 
his administrative appointments. He has broad interests in liberal 
arts education, the quality of undergraduate education, and the role 
of academic advising. Laff has published and presented widely in 
the area of student engagement and success. He is the founder and 
president of the Association of Advisors of English and has been 
nominated as teacher and advisor of the year at the University of 
Illinois, St. Lawrence University, University of Northern Colorado, 

Plorida Atlantic University, and Weber State University. 

Lana Low is the former vice president for retention and assess- Lana Low 
ment services at Noel-Levitz. She has consulted with more than 
100 colleges and universities and is a frequent presenter at national 
and regional conferences and winner of awards for excellence in 
teaching, research, and academic advising. Low's background in- 
cludes senior administrative appointments, direct responsibility for 
student services and assessment, and college teaching. She served 
the University of Virginia at Wise for 19 years before joining Noel- 
Levitz. She received her PhD in educational research and evaluation 
from Virginia Polytechnic Institute and State University. She has 
extensive experience analyzing data on first-year students, including 
developing predictive retention models. 



David A. Lutz, PhD, has worked in higher education since the 1960s David A. Lutz 
with a varied career in teaching, administration, and working for a 
nonprofit testing organization. He has taught at Drew University, 

Upsala College, Bucknell University, and Lycoming College. He was 
the dean and director of assessment for the State of Connecticut's 
nontraditional college, now known as Charter Oak College. He joined 
ACT, Inc. in 1980 and served as the national director of postsecondary 
services at ACT. He played a major role in the outcomes assessment 
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area, including the development and use of the Collegiate Assess- 
ment of Academic Proficiency (CAAP) by more 600 colleges and 
universities since CAAP's inception. He currently is retired and is 
working as a consultant. 

Marla Mamrick Marla Mamrick is a master's student in higher education and student 
affairs with an emphasis higher education administration. Cur- 
rently, she works for the National Resource Center for The First- Year 
Experience and Students in Transition as a graduate assistant for 
research and project development. Marla has been closely involved 
in the development of a course evaluation for University 201 - Fun- 
damentals of Inquiry and assists the Center with other research 
initiatives. Prior to attending graduate school, she worked for the 
Office of the University Registrar at The Ohio State University for 
five years. Her research interests include survey development and 
data analysis. 

Kay M. McClenney Kay McClenney is director of the Community College Survey of 
Student Engagement and an adjunct faculty member in the Com- 
munity College Leadership Program (CCLP) at the University of 
Texas at Austin. She also directs the Ford Foundation's coordinating 
team for the national Community College Bridges to Opportunity 
Initiative and the MetLife Foundation's national student retention 
project. She is senior associate with The Pew Forum on Undergradu- 
ate Learning and a Distinguished Senior Fellow at the Education 
Commission of the States (ECS), where she served as vice president 
and chief operating officer from 1990 to 2000. A frequent keynote 
speaker, McClenney has also authored numerous publications on 
education issues, strategic planning, accountability, and assessment. 
She earned her PhD in educational administration from the Com- 
munity College Leadership Program at the University of Texas at 
Austin, and she has been named a Distinguished Graduate of that 
program. Her previous degrees include a BA from Trinity University 
and an MA in psychology from Texas Christian University. 

Barbara J. Millis Barbara J. Millis has been a frequent presenter at professional con- 
ferences and meetings such as the American Association for Higher 
Education, the Lilly Teaching Conference, the Council of Indepen- 
dent Colleges, and for a variety of colleges and universities. She has 
published on such varied topics as faculty development, learning 
communities, faculty and student portfolios, mentoring, TQM, 
syllabus construction, and peer classroom observation to improve 
teaching. She co-authored, with Philip Cottell, the 1998 book Coop- 
erative Learning for Higher Education Faculty, originally from Oryx 
Press, and now available through Greenwood Press. Millis holds a 
PhD in English literature from Florida State University, is director of 
faculty development at the U.S. Air Force Academy, and was previ- 
ously the assistant dean of faculty development at the University of 
Maryland University College. 
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Libby V. Morris is associate professor and graduate coordinator in Libby V. Morris 
the Institute of Higher Education at the University of Georgia. Morris 
holds a PhD from the University of North Carolina-Chapel Hill. Her 
teaching and research interests include evaluation and assessment, 
academic programs, and distance education. She is currently the 
principal investigator for a multi-year grant to investigate teaching 
and learning online in collaboration with the Advanced Learning 
Technologies unit of the University System of Georgia Board of Re- 
gents. She is editor of Innovative Higher Education, a peer-reviewed, 
international journal focusing on innovations in post-secondary 
education. She is the author or co-author of several books, including 
Multiculturalism in Academe, The Reference Book on Regional Well-Being, 
and the Southern Black Belt-A National Perspective. 

Patricia H. Murrell serves as director of the Center for the Study of Patricia H. Murrell 
Higher Education and professor of higher and adult education at the 
University of Memphis. Since 1994, Murrell has served as director 
of the Community College Student Experiences Questionnaire (CC- 
SEQ) Research Project. She has received the Distinguished Teacher 
Service Award from the University of Memphis, the Crader Award 
from the College of Education, and has twice been cited for supe- 
rior performance in university research. She frequently serves as a 
consultant to colleges, professional organizations, nonprofit groups, 
and corporations and has written more than 50 articles, books, and 
book chapters on learning styles, outcomes assessment, and adult 
development theory in professional performance. Murrell received 
her doctor of education degree from the University of Mississippi. 

David R. Palmer is associate chief financial officer for the Texas Health David R. Palmer 
and Human Services Commission. He has a doctorate in psychomet- 
rics and testing from the University of Texas at Austin. He is also 
chief psychometrician for the Cognitive Learning Strategies Project 
at the University of Texas at Austin and an author of four major as- 
sessment instruments, including the PEEK and LASSI. 



Jerry Pattengale speaks regularly on campuses and at national confer- Jerry Pattengale 
ences and serves on the National Advisory Board for The National 
Resource Center for The First-Year Experience and Students in Transi- 
tion (2003-2006). He co-edited Visible Solutions for Invisible Students: 

Helping Sophomores Succeed (University of South Carolina, 2000) and 
has written articles and developed tools on motivational theory and 
on creative funding solutions for student success initiatives. In ad- 
dition to teaching awards and an NEH fellowship to Greece, he 
was recognized as an Outstanding First- Year Student Advocate by 
Houghton Mifflin and the National Resource Center in 2000. He is a 
professor and an assistant vice president at his alma mater, Indiana 
Wesleyan University where he also oversees and teaches annually 
in the first-year seminar program. He received a doctorate in ancient 
history from Miami University (OH) and holds masters degrees from 
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Miami (Tudor-Stuart history) and Wheaton Graduate School (inter- 
personal development). Pattengale's current research is focused on 
understanding the Millennial, and his student motivation text is in 
press with Mc-Graw Hill. 

Judy Patton Judy Patton is director of university studies at Portland State 
University (PSU). She is a National Fellow with the National 
Learning Communities Project housed at The Washington Center 
for Improving the Quality of Undergraduate Education. Pat- 
ton was a member of the Kellogg Forum of Higher Education 
Transformation and was a member of the Civic Learning Clus- 
ter, a 10-institution collaboration led by NERCHE, that resulted 
from that work. She was the project director for the Quality 
Assurance Collaborative and for the Restructuring for Urban 
Student Success Project with PSU, Temple University and In- 
diana University Purdue University at Indianapolis. She has 
published in the areas of general education, higher education 
reform, learning communities, community-based learning, and 
assessment. She received her BA in dance at the University of 
California, Santa Barbara and a masters in interdisciplinary 
study from Reed College. 

Karen Paulson Karen Paulson is a senior associate at the National Center for 
Higher Education Management Systems (NCHEMS). A member 
of the staff since 1996, she focuses on curricular, assessment, fac- 
ulty, and distance-learning issues as well as state postsecondary 
needs assessments. During 2001-2002, she worked on behalf of 
NCHEMS, in collaboration with the Policy Center on the First 
Year of College, on a major initiative called the "Data Audit and 
Analysis Project." Based on her involvement with the project, 
she authored the Data Audit and Analysis Toolkit to Support As- 
sessment of the First College Year. She conducts evaluations for 
several grants in addition to working on activities to ensure 
appropriate data are available for state-level higher education 
policy making. She was a program associate with the National 
Center for Public Policy and Higher Education for 2001-2002. 

John Pryor John Pryor is the director of undergraduate evaluation and re- 
search at Dartmouth College in the division of student affairs. 
He has extensive experience in surveying college students about 
risky behaviors, as well as in the use of the World Wide Web as 
a surveying tool. 

Marie Revak Marie Revak is the coordinator of professional learning services 
for the Lewis-Palmer School District in Monument, Colorado and 
the former director of academic assessment and associate pro- 
fessor of mathematical sciences at the U.S. Air Force Academy. 
She has a BA in mathematics from Rutgers University, an MA in 
mathematics education from Rowan University (formerly Glass- 
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boro State College), and a PhD in mathematics education from 
the Florida Institute of Technology. Her areas of interest include 
mathematics education, student ratings, multiple-choice testing, 
alternative assessment, and faculty and staff development. 

Beth Richter, senior director of retention products at Noel-Levitz, Beth Richter 
works directly with colleges and universities nationwide in student 
retention. Offering 16 years of experience, Richter provides service 
and leadership to educators seeking to enhance student success and 
improve student retention through the Retention Management Sys- 
tem™ and its College Student Inventory. She is co-author of. Strategic 
Moves for Retention Success, with Lee Noel and Randi Levitz, pub- 
lished in the winter 1999 edition of New Directions for Higher Education. 

Richter has an MA in teaching English, tutors international students, 
and is finalizing coursework toward a PhD in social foundations of 
education at the University of Iowa. 

Linda J. Sax is associate professor in-residence, director of the Coop- Linda J. Sax 
erative Institutional Research Program (CIRP), and associate director 
of the Higher Education Research Institute at UCLA. In that capacity, 
she is responsible for CIRP's annual survey of incoming first-year 
students, two annual surveys of continuing college students, and 
a triennial survey of college faculty. The recipient of the 1999 Early 
Career Award from the Association for the Study of Higher Educa- 
tion, Sax's research focuses on gender differences in college student 
development, specifically how institutional characteristics, peer and 
faculty environments, and forms of student involvement may dif- 
ferentially affect male and female college students. 

Debora Scheffel is currently the director for university assessment Debora L. Scheffel 
at the University of Northern Colorado in Greeley, Colorado. She 
has been a visiting scholar at the United States Air Force Academy, 

Center for Educational Excellence and has presented and published 
in the area of academic assessment. Scheffel received her PhD from 
Northwestern University, Evanston, IL, in communication sciences 
and disorders and completed a post-doctoral fellowship at the Uni- 
versity of California, San Diego, in cognitive psychology. Her in- 
terests include brain-behavior relationships and learning, classroom 
and program assessment, and reform in higher education. 

Michael J. Siegel serves as research fellow with the Policy Center on Michael J. Siegel 

the First Year of College, a national level educational policy center 

supported by The Atlantic Philanthropies and Lumina Foundation 

for Education. He is responsible for a wide range of writing projects 

and research initiatives aimed at improving the first college year. 

Currently, he is coordinating the Phase II process for affiliate institu- 
tions as part of the Center's Foundations of Excellence™ in the First 
College Year project. He also serves as the manager and editor of the 
Policy Center's First- Year Assessment Listserv (FYA-List), recruiting 
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and working with authors to post invited essays on assessment 
and other topics. Siegel holds a PhD in higher education and a 
minor in anthropology from Indiana University His book. Primer 
on Assessment of the First College Year, was published by the Policy 
Center in 2003. 

Tracy L. Skipper Tracy L. Skipper is editorial projects coordinator for the National 
Resource Center for The First- Year Experience and Students in 
Transition at the University of South Carolina. Prior to her work 
at the Center, she served as director of residence life and judicial 
affairs at Shorter College in Rome, Georgia, where her duties 
included teaching in the college's first-year seminar program 
and serving as an academic advisor for first-year students. She 
also served as director of student activities and residence life at 
Wesleyan College. Skipper teaches first-year English and Uni- 
versity 101 at USC. She holds a bachelor's degree in psychology 
from USC, a master's degree in higher education from Florida 
State University, and a master's in American literature from USC. 
She is currently pursuing a doctorate in rhetoric and composition 
and has a research interest in the teaching of writing in first-year 
seminars. 

Linda Suskie Linda Suskie is director of assessment at Towson University, past 
director of the American Association for Higher Education's As- 
sessment Forum, and past fellow at the Middle States Commis- 
sion on Higher Education. Her higher education experience also 
includes work in institutional research and strategic planning and 
teaching graduate courses in assessment and research methods 
and undergraduate courses in writing, statistics, and develop- 
mental mathematics. Suskie has spoken, consulted, written, and 
presented workshops on a broad variety of assessment topics. 
Her latest book. Assessment of Student Learning: A Common Sense 
Guide, is slated for publication by Anker in early 2004. 

Randy L. Swing Randy L. Swing serves as co-director and senior scholar of the 
Policy Center on the First Year of College, located at Brevard 
College in Brevard, North Carolina. During the Center's first 
three years, his work focused on developing and disseminating 
new tools and techniques for evaluating the efficacy of first-year 
programs. His leadership roles included contributions to two 
national surveys of first-year students. Your First College Year 
(YFCY) and the First-Year Initiative (FYI) benchmarking survey. 
Until 1999, Swing worked for 20 years in various first-year pro- 
grams at Appalachian State University. Most recently, as founding 
director of the Assessment Office he developed and initiated a 
longitudinal, campus-wide assessment program with focus on 
learning outcomes. Prior to earning a doctoral degree in higher 
education from the University of Georgia, he earned his MA and 
EdS degrees in student development from Appalachian State 
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University and a BA in psychology from the University of North 
Carolina - Charlotte. He serves on the Review Board for the Journal 
on Excellence in College Teaching and Innovative Higher Education and 
is a fellow of the National Resource Center for The First- Year Expe- 
rience and Students in Transition. Swing's current work includes 
a large-scale project to establish "Foundations of Excellence in the 
First College Year" with the Council for Independent Colleges and 
the American Association for State Colleges and Universities. 

John Ward has served as a vice president at the Clearinghouse since 
it was formed in 1993 and has more than 30 years experience in 
strategic planning, product development, client service, and sales. 
Previously, he led the team that grew Clearinghouse participation to 
include 2,700 colleges and universities. He also directed corporate 
communication activities and the launch of several higher education 
services. Ward is a frequent speaker at conferences geared to college 
financial aid managers, registrars, admissions officers, and higher 
education loan program directors. Before joining the Clearinghouse, 
he held several positions at Sallie Mae, including managing strategic 
planning, product development, client service, and regional banking. 
Prior to his career in student finance, he was in commercial banking. 
Ward has a BA from Amherst College and an MBA from Syracuse 
University. 

Claire Ellen Weinstein is a professor in the Department of Educational 
Psychology and Chair of the Doctoral Concentration in Learning, 
Cognition, and Instruction at the University of Texas at Austin. She 
is also the director of the Cognitive Learning Strategies Research 
Project. Weinstein was recently inducted as a fellow of the American 
Council of Developmental Education Associations and is a member of 
the governing council of the American Psychological Association. She 
has more than 130 publications, including an assessment instrument 
called the Learning and Study Strategies Inventory (LASSI) that is 
used in more than 75% of the colleges and universities in the United 
States and has been translated into more than 30 languages. 

Jean Yerian began work at Virginia Commonwealth University in 
1979. She served as director of the career center, associate dean of 
student affairs, and assistant to the vice provost for academic affairs 
before becoming director of assessment. During a recent year's leave 
of absence, she worked with Zayed University faculty in Dubai and 
Abu Dhabi, United Arab Emirates, shaping a learning-outcomes- 
based curriculum for women leaders. 

Lissa J. Yogan, PhD is currently the chair of the department of sociol- 
ogy and criminology at Valparaiso University. She received her doc- 
torate in sociology from the University of Notre Dame. She regularly 
teaches courses on research methods, social stratification, gender, 
and cultural issues. Her research has looked at gender effects in 
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school violence and the relationship between teen dating and critical 
thinking. Her work in this volume arises from her interest in inter- 
disciplinary teaching and faculty/ administrator collaboration. 



Titles of Interest to Assessment Professionals from 
the National Resource Center for The First-Year Experience & 
Students in Transition 



Monograph 33. Proving and Improving: Strategies for Assessing the First College Year. Randy L. 
Swing, Editor. Drawn from the First -Year Assessment Listserv, which is hosted by the Policy Center 
on the First Year of College and the National Resource Center, this collection provides essays by the 
nation’s best theorists and practitioners of first -year college assessment. Contributors outline the essen- 
tials of effective assessment efforts, provide a philosophical rationale for those essentials, describe meth- 
ods and strategies for assessment, and provide examples designed for institutions and specific programs. 
Written from the perspective of practitioners in a wide range of disciplines and organizational structures, 
these essays are accessible and valuable to the novice and veteran practitioner alike. 1 40 pages. ISBN 
1-889271-37-3. $20.00. 

Primer for Research on the College Student Experience. Dorothy S. Fidler and Jean M. Henscheid. 
This booklet gives an overview of strategies for designing, conducting, and publishing research on the col- 
lege student experience. 68 pages. $ 1 5.00. 

Guidelines for Evaluating The First-Year Experience at Two-Year Colleges and Guidelines for Evalu- 
ating The First-Year Experience at Four-Year Colleges. 2nd Edition. John N. Gardner, Betsy O. 
Barefoot, and Randy L. Swing. These guidelines provide institutions with a framework for assessing the 
first-year experience on their own campuses. Originally developed by John Gardner in 1 990, Guidelines 
for Evaluating The First-Year Experience has been revised and updated by the staff of the Policy Center on 
the First Year of College with input from 65 institutions participating in five regional consortia focused on 
first-year assessment issues. Guidelines is now available in two special editions, one focusing on two-year 
institutions and one focusing on four-year institutions. If you are looking for a way to evaluate the first- 
year experience on your campus, make Guidelines your starting point! 24 pages. $7.00 each/$5.00 each 
when sold in units of 10. 

Annotated Bibliography on Assessment of the First College Year. Marc Cutright and Michelle Ro- 
dems. An annotated bibliography concentrated on the evaluation of programs for first -year students and 
the improvement of those programs through evaluation. The bibliography focuses on print information 
published in the past decade. Both four-year and two-year literature is considered. 79 pages. ISBN 0- 
9726527-2-8. $20.00. 

Primer on Assessment of the First College Year. Michael J. Siegel. This primer is a guidebook on as- 
sessment of the first college year. It is intended for beginning as well as veteran practitioners who are 
responsible for a wide range of assessment activities related to evaluation the learning experience and 
outcomes of first-year college and university students. It probes key campus issues related to first-year 
assessment and offers a comprehensive overview of first-year assessment strategies, methods, and tools. 

1 22 pages. ISBN 0-9726527-0- 1 . $20.00. 

Data Audit and Analysis Toolkit to Support Assessment of the First College Year. Karen Paulson. A 
process for identifying and using information resources in existence on campuses in assessment practice. 
Technical Manual, Administrative Rationale, and CD-ROM are packaged in a three-ring binder. A joint 
project of the Policy Center on the First Year of College and the National Center for Higher Education 
Management Systems. $50.00. 



Use the order form on the next page to order any of these titles from 
the National Resource Center. 
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from the National Resource Center for The First-Year Experience & 
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